-- Logs begin at Tue 2022-09-13 14:49:35 UTC, end at Tue 2022-09-13 15:50:49 UTC. -- Sep 13 15:15:19.114809 dv-cinder-857434-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 13 15:15:19.123804 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 13 15:15:19.125986 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Sep 13 15:15:19 2022] *** Sep 13 15:15:19.125986 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 13 15:15:19.125986 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 13 15:15:19.125986 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: nodename: dv-cinder-857434-2-iscsi Sep 13 15:15:19.125986 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: machine: x86_64 Sep 13 15:15:19.125986 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: clock source: unix Sep 13 15:15:19.125986 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: pcre jit disabled Sep 13 15:15:19.125986 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: detected number of CPU cores: 4 Sep 13 15:15:19.125986 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: current working directory: / Sep 13 15:15:19.125986 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: detected binary path: /usr/bin/uwsgi-core Sep 13 15:15:19.126558 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: your processes number limit is 47680 Sep 13 15:15:19.126558 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: your memory page size is 4096 bytes Sep 13 15:15:19.126558 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: detected max file descriptor number: 2048 Sep 13 15:15:19.126558 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: lock engine: pthread robust mutexes Sep 13 15:15:19.126558 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: thunder lock: enabled Sep 13 15:15:19.126558 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 13 15:15:19.126558 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 13 15:15:19.164921 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: Python main interpreter initialized at 0x5576042989c0 Sep 13 15:15:19.164921 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: python threads support enabled Sep 13 15:15:19.164921 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: your server socket listen backlog is limited to 100 connections Sep 13 15:15:19.164921 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: your mercy for graceful operations on workers is 90 seconds Sep 13 15:15:19.165277 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: mapped 403077 bytes (393 KB) for 2 cores Sep 13 15:15:19.165358 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: *** Operational MODE: preforking *** Sep 13 15:15:19.165615 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: *** uWSGI is running in multiple interpreter mode *** Sep 13 15:15:19.165615 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: spawned uWSGI master process (pid: 81442) Sep 13 15:15:19.165859 dv-cinder-857434-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 13 15:15:19.166950 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: spawned uWSGI worker 1 (pid: 81454, cores: 1) Sep 13 15:15:19.166950 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: spawned uWSGI worker 2 (pid: 81455, cores: 1) Sep 13 15:15:19.166950 dv-cinder-857434-2-iscsi devstack@keystone.service[81442]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 13 15:15:23.544798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 15:15:23.544798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: from cryptography.utils import int_from_bytes Sep 13 15:15:23.545709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 15:15:23.545709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: from cryptography.utils import int_from_bytes Sep 13 15:15:23.545830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 15:15:23.545830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: from cryptography.utils import int_from_bytes Sep 13 15:15:23.546278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 13 15:15:23.546278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: from cryptography.utils import int_from_bytes Sep 13 15:15:23.594389 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 13 15:15:23.594568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 13 15:15:23.594752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] command line args: [] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 13 15:15:23.594897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 13 15:15:23.594897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 13 15:15:23.595034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 13 15:15:23.595112 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] command line args: [] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 13 15:15:23.595200 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ================================================================================ {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 13 15:15:23.595387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 13 15:15:23.595512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] admin_token = **** {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.595597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ================================================================================ {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 13 15:15:23.595667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] config_dir = [] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.595812 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] admin_token = **** {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.595937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.595937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] config_source = [] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.596090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] config_dir = [] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.596167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] debug = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.596245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.596328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.596640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] config_source = [] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.596640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] debug = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.596759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.596822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.596921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.596986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.597042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.597098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.597335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.597431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.597494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] list_limit = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.597629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.597706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] log_config_append = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.597829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] list_limit = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.597912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.597981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] log_config_append = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.598041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] log_dir = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.598303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.598393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] log_file = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.598461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] log_dir = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.598525 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.598592 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] log_file = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.598650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.598706 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.598872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.598950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.599103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.599206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.599281 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.599427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.599523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.599597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.599665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.599782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.599872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.600031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.600163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.600261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.600350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.600494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.600616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.600692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.600761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.600948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.601086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.601173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.601247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.601417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.601539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.601628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.601928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.602060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.16/identity {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.602174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] publish_errors = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.602361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.602524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.16/identity {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.602606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.602678 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] publish_errors = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.602802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.602889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.602971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.603042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.603157 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.603253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.603412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] standard_threads = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.603544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.603629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.603708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.603853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] standard_threads = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] use_journal = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] use_json = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] use_stderr = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] use_journal = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604783 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] use_syslog = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] use_json = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.604922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.605003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] use_stderr = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.605079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.605150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] use_syslog = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.605290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.605421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 13 15:15:23.605539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.605637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.605804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.605927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.606015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.606090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.606273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.606381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.606545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.606668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.606746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.606828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.607010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.607150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.607238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.607310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.607473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.607596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.607684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.607994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.608087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.608160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.608237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.608315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.608387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.608468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.608650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.608734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.608866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.608950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.609120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.609216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.609370 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.609452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.609606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.609686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.609834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.609998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.610073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.610172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.610356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.610437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.610604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.610693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.610848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.610930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.611073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.611154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.611286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.611419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.611508 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.611934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.612083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.612170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.612325 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.612423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.612560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.612662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.612785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.612868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.613010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.613090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.613250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.613328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.613480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.613555 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.613711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.613789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.613953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.614030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.614201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.614291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.614432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.614511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.614661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.614744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.614885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.614962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.615110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.615189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.615335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.615411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.615558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.615635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.615787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.615867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.616001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.616081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.616236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.616313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.616467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.616545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.616692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.616770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.616921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.616998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.617150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.617228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.617386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.617462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.617608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.617685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.617976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.618063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.618223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.618296 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.618367 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.618428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.618551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.618642 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.618756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.618870 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.618986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.619090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.619209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.619288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.619440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.619556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.619652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.619727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.619885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.619978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.620129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.620207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.620357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.620451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.620586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.620674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.620818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.620896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.621050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.621130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.621287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.621380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.621524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.621602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.621766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.621850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.622007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.622094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.622270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.622361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.622502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.622623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.622735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.622822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.622968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.623052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.623202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.623280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.623467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.623594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.623706 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.623783 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.623952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.624196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.624273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.624435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.624514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.624686 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.624763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.624923 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.624999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.625156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.625235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.625390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.625469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.625626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.625704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.625860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.625938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.626126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.626250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.626373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.626475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.626609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.626714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.626845 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.626949 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.627082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.627183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.627355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.627460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.627556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.627657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.627756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.627867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.627973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.628077 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.628175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.628276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.628399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.628503 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.628618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.628724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.628848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.628951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.629087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.629190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.629319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.629431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.629565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.629668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.629798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.629900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.630052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.630191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.630337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.630450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.630724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.630833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.630963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.631071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] auth.external = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.631170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] auth.external = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.631274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.631373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.631469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.631552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.631632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.631766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.631856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] auth.password = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.631998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] auth.password = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.632076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] auth.token = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.632236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] auth.token = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.632315 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.632476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.632564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.632713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.632792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.632940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.633018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.633180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.633266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.633421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.633499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.633651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.633729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.633883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] credential.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.633970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.634155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] credential.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.634239 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.634381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.634459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.634628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.634715 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.634866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.634945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.635107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.635185 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.635340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.635427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.635573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.635652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.635822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.635900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.636052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.636142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.636300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.636377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.636554 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.636633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.636814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.636901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.637073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.637158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.637310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.637387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.637563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.637654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.637795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] federation.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.637874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.638026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.638120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] federation.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.638288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.638378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.638535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.638613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.638764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.638842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.639001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.639137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.639345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.639463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.639584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.639689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.639840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.639948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.640117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.640227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.640332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.640431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.640562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.640668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.640821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.640926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.641042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.641141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.641323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.641425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.641521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.641617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.641713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.641815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.641936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.642174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.642282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.642380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.642476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.642573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.642682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.642783 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.642904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.643020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.643165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.643285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.643409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.643525 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.643639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.643752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.643865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.643978 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.644099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.644217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.644337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.644453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.644567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.645153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.645287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.645368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.645429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.645503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.645503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.645629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.645747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.645823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.645978 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646041 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646508 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646845 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.646994 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.647066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.647194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.647288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.647435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.647517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.647674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.647757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.647936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.648076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.648159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.648232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.648405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.648531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.648611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.648702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.648855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.648981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.649060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.649143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.649304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.649430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.649510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.649584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.649749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.649875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.649968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.650046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.650232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.650357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.650589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.650732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.650826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.650903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.651057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.651183 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.651262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.651335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.651521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.651647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.651727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.651801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.651989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.652116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.652211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.652289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.652440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.652580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.652664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.652737 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.652878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.652963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.653115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.653198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.653343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.653425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.653588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.653672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.653832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.653916 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.654073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.654242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.654335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.654413 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.654543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.654627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.654812 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.654894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.655045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.655160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.655278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.655355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.655494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.655576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.655720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.655803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.655971 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.656056 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.656212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.656294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.656455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.656539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.656697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.656780 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.656919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.657003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.657142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.657221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.657374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.657457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.657606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.657685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.657839 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.658071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.658203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.658328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.658406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.658572 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.658653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.658806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.658885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.659046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.659131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.659274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.659353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.659515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.659593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.659747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.659825 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.659979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.660059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.660212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.660291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.660447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.660524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.660683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.660763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.660911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.660991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.661146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.661225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.661383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] resource.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.661461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.661622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] resource.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.661700 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.661867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.661945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.662115 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.662209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.662358 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.662438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.662610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.662688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.662849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.662928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.663084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.663163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.663318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.663396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.663545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.663621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.663772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] role.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.663852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.664006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] role.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.664093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] role.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.664230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.664308 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] role.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.664476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.664554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.664720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.664806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.664964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.665044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.665208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.665287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.665451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.665539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.665695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.665773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.665934 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.666015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.666199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.666297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.666428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.666508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.666667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.666744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.666907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.666994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.667150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.667230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.667391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.667470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.667639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.667800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.668037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.668114 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.668269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.668347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.668520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.668609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.668766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.668844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.669012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.669090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.669241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.669331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.669481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.669566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.669726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.669804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.669962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.670042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.670215 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.670392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.670522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.670625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.670716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.670793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.670937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.671112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.671191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.671369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.671457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.671615 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.671695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.671849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.671927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.672081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.672168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.672309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.672388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.672543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] token.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.672622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] token.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.672778 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.672865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.673025 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.673103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.673257 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.673336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.673511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.673597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.673754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.673832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.673986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.674065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.674248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.674341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.674491 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.674570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.674737 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.674815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.674966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.675053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.675204 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.675282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.675432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.675510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.675677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.675764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.675952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.676030 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.676178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.676265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.676415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.676504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.676677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.676755 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.676935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.connection = **** {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.677015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.connection = **** {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.677177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.677265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.677421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.677500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.677651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.677730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.677889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.677967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.678146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.678228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.678369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.678452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.678599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.678679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.678830 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.678908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.679073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.679153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.679307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.679385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.679548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.679626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.679778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.679859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.680022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.680102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.680252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.680331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.680492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.680572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.680718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.680797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.680953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.681031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.681181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.681261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 13 15:15:23.681334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81454) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 13 15:15:23.681406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81455) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 13 15:15:23.710854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.711033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.711104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.711182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.711284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.711368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.719726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.719824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.747811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.748058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.748244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.748531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.801501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.812292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.849607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.849813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.860031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:23.860269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.008226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.008436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.008625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.008872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.009022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.009110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.009185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.009258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.009435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.009621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.009862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.010069 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.032305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.032520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.032733 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.032881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.032955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.033031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.033160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.033283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.033371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.033452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.033567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.033655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.033787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.034019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.044710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.044881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.057478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.057613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.057780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.057893 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.058040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.058191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.058261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.058338 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.058465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.058557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.058703 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.058835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81455) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.058903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.059106 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81454) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 13 15:15:24.075428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.075643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 15:15:24.075739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.075945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 15:15:24.076068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.076157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 15:15:24.076234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.076416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.076622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 13 15:15:24.076710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.076787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.076940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.077073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.077216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.077340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.077633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.086913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.087202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.087328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.087445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.087538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.087751 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.087878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.088034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.094999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.095363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.097915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.098230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.098355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.098513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.098640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.098792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.098939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.099040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.099130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.099339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.099464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.099548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.099628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.099842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.099969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.100111 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.100248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.100325 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.100400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.100626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.100751 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.100895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.124102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.124253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.125559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.125684 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.125839 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.125982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.130556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.130798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.130926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.131112 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.131255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.131587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.137472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.137766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.142287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.142693 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.143577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.143872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.144161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.144450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.144732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.145016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.145272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.145542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.151317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.151730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.152130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.152520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.152904 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.153291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.153665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.154045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.161076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.161367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.165241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.168501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.168805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.173056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.175188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.175487 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.176180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.179314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.179461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.179695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.179962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.180271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.180563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.180837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.182505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.182797 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.187142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.190217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.193715 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.194010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.194314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.194595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.194876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.195152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.196617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.196912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.197181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.197448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.204630 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.206222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.207750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.208034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.210903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.211149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.211337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.211479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.211560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.211635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.211709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.211926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.212111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.218810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.220332 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.221846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.222136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.223579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.223866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.224154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.224436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.224769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.224909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.225061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.225189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.225336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.225478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.225586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.225996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.237566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.237851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.238129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.238440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.238788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.239063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.239337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.241263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.244234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.247181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.248682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.250160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.250469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.250878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.255053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.257271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.257956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.258763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.260877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.261662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.262059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.262342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.262468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.262638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.263792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.264368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.264607 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.270584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.270878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.271000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.271156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.272432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.275927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.276540 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.277012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.277470 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.278090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81455) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.278403 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.278682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.278959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.279232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.279510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.280048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.280333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.280615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.280901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.281172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.281435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.281705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.281993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81455) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.289097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.289394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.289667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.296809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81454) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 13 15:15:24.297088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.297367 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.297640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.297919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.298222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.298495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.298768 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.299032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.299315 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.299564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.299830 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.300089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.300359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81454) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 13 15:15:24.316263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5576042989c0 pid: 81455 (default app) Sep 13 15:15:24.319593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64b8089-8bfb-4383-8fd0-212d5b5eaba9 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:24.319889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64b8089-8bfb-4383-8fd0-212d5b5eaba9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:24.320164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64b8089-8bfb-4383-8fd0-212d5b5eaba9 None None] PATH_INFO: `/v3` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:24.324383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1/1] 10.222.0.16 () {50 vars in 791 bytes} [Tue Sep 13 15:15:24 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:24.334573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5576042989c0 pid: 81454 (default app) Sep 13 15:15:31.185455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3529ce1b-31b4-4bf2-bbcd-0aeeeda5ff6f None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:31.185983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3529ce1b-31b4-4bf2-bbcd-0aeeeda5ff6f None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:31.185983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3529ce1b-31b4-4bf2-bbcd-0aeeeda5ff6f None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:31.188003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 2/2] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:31.200404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3144070b-8298-438b-96e6-981e8cef81db None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:31.200903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3144070b-8298-438b-96e6-981e8cef81db None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:31.200996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3144070b-8298-438b-96e6-981e8cef81db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:31.283198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: INFO dbcounter [None req-3144070b-8298-438b-96e6-981e8cef81db None None] Registered counter for database keystone Sep 13 15:15:31.304601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG oslo_db.sqlalchemy.engines [None req-3144070b-8298-438b-96e6-981e8cef81db None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81454) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 13 15:15:31.311430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writer thread running {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 13 15:15:31.491530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG passlib.handlers.bcrypt [None req-3144070b-8298-438b-96e6-981e8cef81db None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81454) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 13 15:15:31.491825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG passlib.handlers.bcrypt [None req-3144070b-8298-438b-96e6-981e8cef81db None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81454) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 13 15:15:31.550178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3144070b-8298-438b-96e6-981e8cef81db None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:31.551755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3144070b-8298-438b-96e6-981e8cef81db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:31.692250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1/3] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:31 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 497 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 13 15:15:32.200900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3529ce1b-31b4-4bf2-bbcd-0aeeeda5ff6f None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:32.277828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: INFO dbcounter [None req-3529ce1b-31b4-4bf2-bbcd-0aeeeda5ff6f None None] Registered counter for database keystone Sep 13 15:15:32.303402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG oslo_db.sqlalchemy.engines [None req-3529ce1b-31b4-4bf2-bbcd-0aeeeda5ff6f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81455) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 13 15:15:32.314518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writer thread running {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 13 15:15:32.436728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3529ce1b-31b4-4bf2-bbcd-0aeeeda5ff6f None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:32.442205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ad430e-2008-4067-a9bc-b20d01e2df31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:32.442842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ad430e-2008-4067-a9bc-b20d01e2df31 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:32.443118 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ad430e-2008-4067-a9bc-b20d01e2df31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:32.443393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ad430e-2008-4067-a9bc-b20d01e2df31 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:32.452712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ad430e-2008-4067-a9bc-b20d01e2df31 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:32.504427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: WARNING keystone.common.rbac_enforcer.enforcer [None req-f8ad430e-2008-4067-a9bc-b20d01e2df31 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 13 15:15:32.504783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ad430e-2008-4067-a9bc-b20d01e2df31 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:32.515233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 3/4] 10.222.0.16 () {58 vars in 1324 bytes} [Tue Sep 13 15:15:32 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:15:32.522536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3144070b-8298-438b-96e6-981e8cef81db None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:32.549452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3144070b-8298-438b-96e6-981e8cef81db None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:32.555236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c671ec-2a7f-4b69-b2cc-863d4fb91dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:32.556524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c671ec-2a7f-4b69-b2cc-863d4fb91dc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:32.556801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c671ec-2a7f-4b69-b2cc-863d4fb91dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:32.557072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c671ec-2a7f-4b69-b2cc-863d4fb91dc6 admin admin] PATH_INFO: `/v3/projects/3dcdfc529b6149688ceee6b949a735a1` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:32.558734 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c671ec-2a7f-4b69-b2cc-863d4fb91dc6 admin admin] RBAC: Authorizing `identity:get_project(project_id=3dcdfc529b6149688ceee6b949a735a1)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:32.610659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: WARNING keystone.common.rbac_enforcer.enforcer [None req-81c671ec-2a7f-4b69-b2cc-863d4fb91dc6 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 13 15:15:32.610976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c671ec-2a7f-4b69-b2cc-863d4fb91dc6 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:32.615894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 2/5] 10.222.0.16 () {56 vars in 1207 bytes} [Tue Sep 13 15:15:32 2022] GET /identity/v3/projects/3dcdfc529b6149688ceee6b949a735a1 => generated 344 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:34.745175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43885d9f-b8cf-46a9-a017-a13b5b9cc935 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:34.745712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43885d9f-b8cf-46a9-a017-a13b5b9cc935 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:34.745712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43885d9f-b8cf-46a9-a017-a13b5b9cc935 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:34.749920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 4/6] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:34.756779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bffc028-6d25-4d54-b7b0-9e7484ca77d9 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:34.757494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bffc028-6d25-4d54-b7b0-9e7484ca77d9 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:34.757615 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bffc028-6d25-4d54-b7b0-9e7484ca77d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:34.783210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-1bffc028-6d25-4d54-b7b0-9e7484ca77d9 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:34.784746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1bffc028-6d25-4d54-b7b0-9e7484ca77d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:34.795126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 3/7] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:34 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 13 15:15:35.076797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43885d9f-b8cf-46a9-a017-a13b5b9cc935 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:35.091388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43885d9f-b8cf-46a9-a017-a13b5b9cc935 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:35.096882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d73e9e-2a50-4904-b691-72b0c3c01d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:35.097492 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d73e9e-2a50-4904-b691-72b0c3c01d14 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:35.097769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d73e9e-2a50-4904-b691-72b0c3c01d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:35.098040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d73e9e-2a50-4904-b691-72b0c3c01d14 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:35.106755 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d73e9e-2a50-4904-b691-72b0c3c01d14 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:35.108566 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d73e9e-2a50-4904-b691-72b0c3c01d14 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:35.118736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 5/8] 10.222.0.16 () {58 vars in 1324 bytes} [Tue Sep 13 15:15:35 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:15:35.124835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bffc028-6d25-4d54-b7b0-9e7484ca77d9 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:35.134831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bffc028-6d25-4d54-b7b0-9e7484ca77d9 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:35.141274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b718da31-06ca-49a4-9537-b2f9091eef9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:35.141906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718da31-06ca-49a4-9537-b2f9091eef9e admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:35.142212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718da31-06ca-49a4-9537-b2f9091eef9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:35.142480 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718da31-06ca-49a4-9537-b2f9091eef9e admin admin] PATH_INFO: `/v3/users/d29c3b9b96c24a7daf5c64be528badeb` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:35.144004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b718da31-06ca-49a4-9537-b2f9091eef9e admin admin] RBAC: Authorizing `identity:get_user(user_id=d29c3b9b96c24a7daf5c64be528badeb)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:35.145841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b718da31-06ca-49a4-9537-b2f9091eef9e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:35.147397 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 4/9] 10.222.0.16 () {56 vars in 1198 bytes} [Tue Sep 13 15:15:35 2022] GET /identity/v3/users/d29c3b9b96c24a7daf5c64be528badeb => generated 247 bytes in 23 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:37.390411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efba2b1-af67-4385-9132-cd2dccc04432 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:37.390981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efba2b1-af67-4385-9132-cd2dccc04432 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:37.390981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efba2b1-af67-4385-9132-cd2dccc04432 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:37.392335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 6/10] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:37.398721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e9b9d7-6575-4c91-b4af-81ce3f661b90 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:37.399161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e9b9d7-6575-4c91-b4af-81ce3f661b90 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:37.399257 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e9b9d7-6575-4c91-b4af-81ce3f661b90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:37.400922 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94efddd0-0edb-4c5f-b156-5e72fb1c58ad None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:37.401205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94efddd0-0edb-4c5f-b156-5e72fb1c58ad None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:37.401478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94efddd0-0edb-4c5f-b156-5e72fb1c58ad None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:37.402992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 7/11] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:15:37.408579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64763bf4-3a07-4f64-98c5-3ac0d858098a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:37.408861 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64763bf4-3a07-4f64-98c5-3ac0d858098a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:37.409132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64763bf4-3a07-4f64-98c5-3ac0d858098a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:37.420966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-a6e9b9d7-6575-4c91-b4af-81ce3f661b90 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:37.422345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a6e9b9d7-6575-4c91-b4af-81ce3f661b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:37.453068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG passlib.handlers.bcrypt [None req-64763bf4-3a07-4f64-98c5-3ac0d858098a None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81455) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 13 15:15:37.453468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG passlib.handlers.bcrypt [None req-64763bf4-3a07-4f64-98c5-3ac0d858098a None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81455) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 13 15:15:37.504693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-64763bf4-3a07-4f64-98c5-3ac0d858098a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:37.506383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-64763bf4-3a07-4f64-98c5-3ac0d858098a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:37.509495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 5/12] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:37.607898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 8/13] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:15:37.770918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e9b9d7-6575-4c91-b4af-81ce3f661b90 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:37.799000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:37.878560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64763bf4-3a07-4f64-98c5-3ac0d858098a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:37.884026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e9b9d7-6575-4c91-b4af-81ce3f661b90 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:37.917967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784d4c7e-e21b-4130-b658-2c63c8b00c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:37.918620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784d4c7e-e21b-4130-b658-2c63c8b00c48 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:37.918905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784d4c7e-e21b-4130-b658-2c63c8b00c48 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:37.919181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784d4c7e-e21b-4130-b658-2c63c8b00c48 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:37.979420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64763bf4-3a07-4f64-98c5-3ac0d858098a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:37.981704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:37.985737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784d4c7e-e21b-4130-b658-2c63c8b00c48 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:37.987257 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784d4c7e-e21b-4130-b658-2c63c8b00c48 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:38.011746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:38.019189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958053b9-b7e2-4c39-a519-c8e12710b1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:38.020341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958053b9-b7e2-4c39-a519-c8e12710b1b9 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:38.020624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958053b9-b7e2-4c39-a519-c8e12710b1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:38.020898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958053b9-b7e2-4c39-a519-c8e12710b1b9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:38.032884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958053b9-b7e2-4c39-a519-c8e12710b1b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-958053b9-b7e2-4c39-a519-c8e12710b1b9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:15:38.041455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:38.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:15:38.043347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 9/14] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:15:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:15:38.048910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958053b9-b7e2-4c39-a519-c8e12710b1b9 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:38.092637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 6/15] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:15:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:15:38.098442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784d4c7e-e21b-4130-b658-2c63c8b00c48 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:38.152430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958053b9-b7e2-4c39-a519-c8e12710b1b9 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:38.185794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b5549a-7919-457d-b814-3798dbdf1e65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:38.186515 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b5549a-7919-457d-b814-3798dbdf1e65 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:38.187186 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b5549a-7919-457d-b814-3798dbdf1e65 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:38.187460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b5549a-7919-457d-b814-3798dbdf1e65 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:38.188138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b5549a-7919-457d-b814-3798dbdf1e65 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:38.188442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b5549a-7919-457d-b814-3798dbdf1e65 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:38.189892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b5549a-7919-457d-b814-3798dbdf1e65 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:38.197006 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784d4c7e-e21b-4130-b658-2c63c8b00c48 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:38.201439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 10/16] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:15:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:15:38.207748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b5549a-7919-457d-b814-3798dbdf1e65 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:38.220374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:38.234130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf88b49a-a63e-43d5-9fb1-3a1ec9a69f39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:38.234806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf88b49a-a63e-43d5-9fb1-3a1ec9a69f39 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:38.235085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf88b49a-a63e-43d5-9fb1-3a1ec9a69f39 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:38.235353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf88b49a-a63e-43d5-9fb1-3a1ec9a69f39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:38.236289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:38.245790 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf88b49a-a63e-43d5-9fb1-3a1ec9a69f39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:38.247310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf88b49a-a63e-43d5-9fb1-3a1ec9a69f39 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-bf88b49a-a63e-43d5-9fb1-3a1ec9a69f39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:15:38.266366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:38.267260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:15:38.268331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:15:38.268331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:15:38.268331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:15:38.268331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 7/17] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:15:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:15:38.273577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf88b49a-a63e-43d5-9fb1-3a1ec9a69f39 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:38.314614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b5549a-7919-457d-b814-3798dbdf1e65 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:38.350836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cfca25-0155-476e-b1ee-6d247a39bbf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:38.351522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cfca25-0155-476e-b1ee-6d247a39bbf7 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:38.351801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cfca25-0155-476e-b1ee-6d247a39bbf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:38.352075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cfca25-0155-476e-b1ee-6d247a39bbf7 None admin] PATH_INFO: `/v3/users/d29c3b9b96c24a7daf5c64be528badeb` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:38.353603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cfca25-0155-476e-b1ee-6d247a39bbf7 None admin] RBAC: Authorizing `identity:get_user(user_id=d29c3b9b96c24a7daf5c64be528badeb)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:38.355141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cfca25-0155-476e-b1ee-6d247a39bbf7 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:38.356738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 11/18] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:15:38 2022] GET /identity/v3/users/d29c3b9b96c24a7daf5c64be528badeb => generated 247 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:15:38.362992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cfca25-0155-476e-b1ee-6d247a39bbf7 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:38.377911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf88b49a-a63e-43d5-9fb1-3a1ec9a69f39 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:38.411546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066ab171-cc2b-4a8c-b6ed-062238288f62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:38.412170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066ab171-cc2b-4a8c-b6ed-062238288f62 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:38.412432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066ab171-cc2b-4a8c-b6ed-062238288f62 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:38.412706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066ab171-cc2b-4a8c-b6ed-062238288f62 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:38.413349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066ab171-cc2b-4a8c-b6ed-062238288f62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:38.413650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066ab171-cc2b-4a8c-b6ed-062238288f62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:38.415095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066ab171-cc2b-4a8c-b6ed-062238288f62 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:38.432594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 8/19] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:15:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:15:38.463547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:38.472441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cfca25-0155-476e-b1ee-6d247a39bbf7 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:38.511431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63281e1-cb20-4acb-ad69-26d942623d2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:38.512086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63281e1-cb20-4acb-ad69-26d942623d2b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:38.512376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63281e1-cb20-4acb-ad69-26d942623d2b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:38.512618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63281e1-cb20-4acb-ad69-26d942623d2b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:38.513852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63281e1-cb20-4acb-ad69-26d942623d2b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:38.516303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63281e1-cb20-4acb-ad69-26d942623d2b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:38.517953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 12/20] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:15:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:15:38.524589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066ab171-cc2b-4a8c-b6ed-062238288f62 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:38.564224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:38.637950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066ab171-cc2b-4a8c-b6ed-062238288f62 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:38.675734 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbf8cd1-c08b-4386-9b56-f27e4cf233ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:38.676356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbf8cd1-c08b-4386-9b56-f27e4cf233ab None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:38.676618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbf8cd1-c08b-4386-9b56-f27e4cf233ab None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:38.676910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbf8cd1-c08b-4386-9b56-f27e4cf233ab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:38.677646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbf8cd1-c08b-4386-9b56-f27e4cf233ab None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.domain.id=default, user.id=d29c3b9b96c24a7daf5c64be528badeb) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:38.677952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbf8cd1-c08b-4386-9b56-f27e4cf233ab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:38.679463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbf8cd1-c08b-4386-9b56-f27e4cf233ab None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:38.687126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 9/21] 10.222.0.16 () {56 vars in 1322 bytes} [Tue Sep 13 15:15:38 2022] GET /identity/v3/role_assignments?user.id=d29c3b9b96c24a7daf5c64be528badeb&scope.domain.id=default&role.id=22819414415942d399db534708fe7103 => generated 233 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:40.830351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbffb62-7ec4-48a3-8518-0b700009d4c7 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:40.830888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbffb62-7ec4-48a3-8518-0b700009d4c7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:40.830888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbffb62-7ec4-48a3-8518-0b700009d4c7 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:40.832210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 13/22] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:40.837568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595a9c33-ad48-4cae-bb58-16ffdfff95d9 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:40.838414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595a9c33-ad48-4cae-bb58-16ffdfff95d9 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:40.838887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595a9c33-ad48-4cae-bb58-16ffdfff95d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:40.867156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-595a9c33-ad48-4cae-bb58-16ffdfff95d9 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:40.868695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-595a9c33-ad48-4cae-bb58-16ffdfff95d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:40.952908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 10/23] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:15:41.313832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda8a3cc-7c9a-4d2c-ae9c-c551d5abc2b6 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:41.314458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda8a3cc-7c9a-4d2c-ae9c-c551d5abc2b6 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:41.314953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda8a3cc-7c9a-4d2c-ae9c-c551d5abc2b6 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:41.316703 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 14/24] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:41.322681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c350a05-5635-415b-91d9-4743fa2a02d1 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:41.323218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c350a05-5635-415b-91d9-4743fa2a02d1 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:41.323677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c350a05-5635-415b-91d9-4743fa2a02d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:41.351408 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2c350a05-5635-415b-91d9-4743fa2a02d1 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:41.352997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2c350a05-5635-415b-91d9-4743fa2a02d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:41.380828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:41.438587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 11/25] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:41.445411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda8a3cc-7c9a-4d2c-ae9c-c551d5abc2b6 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:41.446113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9968b715-d4d7-45db-815b-83c8c55030fb None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:41.446414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9968b715-d4d7-45db-815b-83c8c55030fb None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:41.446690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9968b715-d4d7-45db-815b-83c8c55030fb None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:41.448273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 12/26] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:41.453814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c354c6-6c42-4736-bb76-3a7c172ab81e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:41.454097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c354c6-6c42-4736-bb76-3a7c172ab81e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:41.454414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c354c6-6c42-4736-bb76-3a7c172ab81e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:41.488157 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-f5c354c6-6c42-4736-bb76-3a7c172ab81e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:41.489464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f5c354c6-6c42-4736-bb76-3a7c172ab81e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:41.603541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 13/27] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:15:41.604761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12265320-74fe-4377-bd09-23caf7778e69 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:41.605045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12265320-74fe-4377-bd09-23caf7778e69 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:41.605318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12265320-74fe-4377-bd09-23caf7778e69 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:41.606578 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 14/28] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:41.613010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a822fa-586f-4ae8-a49a-bb2fa77c368f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:41.613290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a822fa-586f-4ae8-a49a-bb2fa77c368f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:41.613561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a822fa-586f-4ae8-a49a-bb2fa77c368f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:41.653176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda8a3cc-7c9a-4d2c-ae9c-c551d5abc2b6 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:41.654081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-84a822fa-586f-4ae8-a49a-bb2fa77c368f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:41.655339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-84a822fa-586f-4ae8-a49a-bb2fa77c368f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:41.724928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6f6e9e-dbc3-4560-83c6-9fcb8a07e8fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:41.725594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6f6e9e-dbc3-4560-83c6-9fcb8a07e8fa None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:41.725876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6f6e9e-dbc3-4560-83c6-9fcb8a07e8fa None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:41.726170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6f6e9e-dbc3-4560-83c6-9fcb8a07e8fa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:41.754323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6f6e9e-dbc3-4560-83c6-9fcb8a07e8fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:41.756219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6f6e9e-dbc3-4560-83c6-9fcb8a07e8fa None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-2c6f6e9e-dbc3-4560-83c6-9fcb8a07e8fa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:15:41.785182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:41.786091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:15:41.786919 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:15:41.786919 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:15:41.786919 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:15:41.789763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 15/29] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:15:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 345 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:15:41.790602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6f6e9e-dbc3-4560-83c6-9fcb8a07e8fa None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:41.802988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 15/30] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:15:41.803859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a822fa-586f-4ae8-a49a-bb2fa77c368f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:41.836194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:41.868852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:41.995013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a822fa-586f-4ae8-a49a-bb2fa77c368f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:42.000160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6f6e9e-dbc3-4560-83c6-9fcb8a07e8fa None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:42.030024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d928e5a0-c6e0-4207-9f3c-07dad6682bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:42.031114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d928e5a0-c6e0-4207-9f3c-07dad6682bdd None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:42.031589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d928e5a0-c6e0-4207-9f3c-07dad6682bdd None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:42.032053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d928e5a0-c6e0-4207-9f3c-07dad6682bdd None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:42.032900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d928e5a0-c6e0-4207-9f3c-07dad6682bdd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:42.033340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d928e5a0-c6e0-4207-9f3c-07dad6682bdd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:42.034937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d928e5a0-c6e0-4207-9f3c-07dad6682bdd None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:42.048290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 16/31] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:15:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:42.053250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d928e5a0-c6e0-4207-9f3c-07dad6682bdd None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:42.081528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d30c5-b94f-447d-8b6a-b5b7ebd86ca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:42.082182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d30c5-b94f-447d-8b6a-b5b7ebd86ca9 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:42.082459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d30c5-b94f-447d-8b6a-b5b7ebd86ca9 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:42.082720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d30c5-b94f-447d-8b6a-b5b7ebd86ca9 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:42.083559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307d30c5-b94f-447d-8b6a-b5b7ebd86ca9 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:42.085085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307d30c5-b94f-447d-8b6a-b5b7ebd86ca9 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:42.142626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:42.182536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d928e5a0-c6e0-4207-9f3c-07dad6682bdd None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:42.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d976f4d2-10b2-4493-866d-e99afd1b17df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:42.242810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d976f4d2-10b2-4493-866d-e99afd1b17df None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:42.243090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d976f4d2-10b2-4493-866d-e99afd1b17df None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:42.243358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d976f4d2-10b2-4493-866d-e99afd1b17df None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:42.256576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 16/32] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Sep 13 15:15:41 2022] POST /identity/v3/roles => generated 225 bytes in 466 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:15:42.257976 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d30c5-b94f-447d-8b6a-b5b7ebd86ca9 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:42.271445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d976f4d2-10b2-4493-866d-e99afd1b17df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-d976f4d2-10b2-4493-866d-e99afd1b17df None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:15:42.283291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:42.284248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:15:42.284967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:15:42.289861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 17/33] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:15:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:15:42.296977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d976f4d2-10b2-4493-866d-e99afd1b17df None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:42.361639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:42.390893 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d30c5-b94f-447d-8b6a-b5b7ebd86ca9 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:42.429758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99a1da6-2a30-45f8-991b-e04f60098cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:42.430816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99a1da6-2a30-45f8-991b-e04f60098cdd None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:42.431827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99a1da6-2a30-45f8-991b-e04f60098cdd None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:42.432262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99a1da6-2a30-45f8-991b-e04f60098cdd None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:42.433263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99a1da6-2a30-45f8-991b-e04f60098cdd None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:42.435239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99a1da6-2a30-45f8-991b-e04f60098cdd None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:42.480248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 17/34] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:15:42 2022] POST /identity/v3/projects => generated 301 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:15:42.481499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99a1da6-2a30-45f8-991b-e04f60098cdd None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:42.550316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d976f4d2-10b2-4493-866d-e99afd1b17df None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:42.615780 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99a1da6-2a30-45f8-991b-e04f60098cdd None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:42.633558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a9a3f6-915a-42ad-8d2b-1d3aff0e45fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:42.634227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a9a3f6-915a-42ad-8d2b-1d3aff0e45fa None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:42.634495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a9a3f6-915a-42ad-8d2b-1d3aff0e45fa None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:42.634765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a9a3f6-915a-42ad-8d2b-1d3aff0e45fa None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:42.635453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a9a3f6-915a-42ad-8d2b-1d3aff0e45fa None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:42.635751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a9a3f6-915a-42ad-8d2b-1d3aff0e45fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:42.637197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a9a3f6-915a-42ad-8d2b-1d3aff0e45fa None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:42.654148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:42.658114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:42.664261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2b112a-3203-4171-928a-87e12aed8fe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:42.664920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2b112a-3203-4171-928a-87e12aed8fe9 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:42.665190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 18/35] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:15:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 369 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:42.665566 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2b112a-3203-4171-928a-87e12aed8fe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:42.665850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2b112a-3203-4171-928a-87e12aed8fe9 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:42.666590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2b112a-3203-4171-928a-87e12aed8fe9 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:42.668176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2b112a-3203-4171-928a-87e12aed8fe9 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:42.681223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a9a3f6-915a-42ad-8d2b-1d3aff0e45fa None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:42.699099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 18/36] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Sep 13 15:15:42 2022] POST /identity/v3/roles => generated 219 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:15:42.816421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a9a3f6-915a-42ad-8d2b-1d3aff0e45fa None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:42.855552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf26084-2f63-4eb4-a4a0-f84e97d20e65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:42.856360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf26084-2f63-4eb4-a4a0-f84e97d20e65 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:42.856643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf26084-2f63-4eb4-a4a0-f84e97d20e65 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:42.856917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf26084-2f63-4eb4-a4a0-f84e97d20e65 None admin] PATH_INFO: `/v3/users/d29c3b9b96c24a7daf5c64be528badeb` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:42.858300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf26084-2f63-4eb4-a4a0-f84e97d20e65 None admin] RBAC: Authorizing `identity:get_user(user_id=d29c3b9b96c24a7daf5c64be528badeb)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:42.859930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf26084-2f63-4eb4-a4a0-f84e97d20e65 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:42.861971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 19/37] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:15:42 2022] GET /identity/v3/users/d29c3b9b96c24a7daf5c64be528badeb => generated 247 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:15:42.868355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2b112a-3203-4171-928a-87e12aed8fe9 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:42.890226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49683233-e5c5-4a84-b352-7f609e0e9b87 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:42.890752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49683233-e5c5-4a84-b352-7f609e0e9b87 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:42.891074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49683233-e5c5-4a84-b352-7f609e0e9b87 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:42.892740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 20/38] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:15:42.899015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61864f45-334e-4eec-88df-4cbc37091a1f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:42.899335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61864f45-334e-4eec-88df-4cbc37091a1f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:42.899571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61864f45-334e-4eec-88df-4cbc37091a1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:42.919050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-61864f45-334e-4eec-88df-4cbc37091a1f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:42.920285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-61864f45-334e-4eec-88df-4cbc37091a1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:43.005918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:43.012131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 21/39] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:15:43.012416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2b112a-3203-4171-928a-87e12aed8fe9 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:43.013521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7921393-4636-4cf2-af9e-2eae9858df70 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:43.013882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7921393-4636-4cf2-af9e-2eae9858df70 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:43.014236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7921393-4636-4cf2-af9e-2eae9858df70 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:43.016214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 22/40] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:43.021600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd7683b-97a9-46c9-94ff-95395de57766 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:43.021877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd7683b-97a9-46c9-94ff-95395de57766 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:43.022213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd7683b-97a9-46c9-94ff-95395de57766 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:43.040462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-1cd7683b-97a9-46c9-94ff-95395de57766 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:43.041633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1cd7683b-97a9-46c9-94ff-95395de57766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:43.062747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673c464b-feaa-4b0b-8647-aa45bc56e41b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:43.063407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673c464b-feaa-4b0b-8647-aa45bc56e41b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:43.063685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673c464b-feaa-4b0b-8647-aa45bc56e41b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:43.063960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673c464b-feaa-4b0b-8647-aa45bc56e41b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:43.065172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673c464b-feaa-4b0b-8647-aa45bc56e41b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:43.066682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673c464b-feaa-4b0b-8647-aa45bc56e41b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:43.073007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 19/41] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:15:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:43.073883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673c464b-feaa-4b0b-8647-aa45bc56e41b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:43.114507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 23/42] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:15:43.134433 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:43.182782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673c464b-feaa-4b0b-8647-aa45bc56e41b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:43.216476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae5f987-3aec-4d8e-a22f-e43df88151a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:43.217142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5f987-3aec-4d8e-a22f-e43df88151a3 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:43.217419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5f987-3aec-4d8e-a22f-e43df88151a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:43.217700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5f987-3aec-4d8e-a22f-e43df88151a3 None admin] PATH_INFO: `/v3/domains/default/users/d29c3b9b96c24a7daf5c64be528badeb/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:43.219755 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae5f987-3aec-4d8e-a22f-e43df88151a3 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=d29c3b9b96c24a7daf5c64be528badeb, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:43.221409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae5f987-3aec-4d8e-a22f-e43df88151a3 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:43.238440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 20/43] 10.222.0.16 () {58 vars in 1361 bytes} [Tue Sep 13 15:15:43 2022] PUT /identity/v3/domains/default/users/d29c3b9b96c24a7daf5c64be528badeb/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:15:43.291857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd7683b-97a9-46c9-94ff-95395de57766 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:43.378219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae5f987-3aec-4d8e-a22f-e43df88151a3 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:43.407274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd7683b-97a9-46c9-94ff-95395de57766 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:43.438552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:43.447664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833c1172-ef50-436e-a4c0-7f8d5c48c050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:43.448297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c1172-ef50-436e-a4c0-7f8d5c48c050 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:43.448577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c1172-ef50-436e-a4c0-7f8d5c48c050 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:43.448995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c1172-ef50-436e-a4c0-7f8d5c48c050 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:43.449992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833c1172-ef50-436e-a4c0-7f8d5c48c050 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:43.451870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833c1172-ef50-436e-a4c0-7f8d5c48c050 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:43.490651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae5f987-3aec-4d8e-a22f-e43df88151a3 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:43.504527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:43.526217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa307bf-398d-4987-a491-a7392c9e95eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:43.526930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa307bf-398d-4987-a491-a7392c9e95eb None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:43.527212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa307bf-398d-4987-a491-a7392c9e95eb None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:43.527531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa307bf-398d-4987-a491-a7392c9e95eb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:43.528764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa307bf-398d-4987-a491-a7392c9e95eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:43.530389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa307bf-398d-4987-a491-a7392c9e95eb None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:43.532300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 21/44] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:15:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:43.538073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa307bf-398d-4987-a491-a7392c9e95eb None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:43.654497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa307bf-398d-4987-a491-a7392c9e95eb None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:43.689189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ab97e9-afed-42d5-a669-edb47315446c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:43.689882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ab97e9-afed-42d5-a669-edb47315446c None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:43.690181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ab97e9-afed-42d5-a669-edb47315446c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:43.690454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ab97e9-afed-42d5-a669-edb47315446c None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:43.691109 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ab97e9-afed-42d5-a669-edb47315446c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:43.692553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ab97e9-afed-42d5-a669-edb47315446c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:43.695582 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 24/45] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Sep 13 15:15:43 2022] POST /identity/v3/roles => generated 223 bytes in 404 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:15:43.727069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 22/46] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:15:43 2022] POST /identity/v3/projects => generated 312 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:15:45.390856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541058c5-1097-412d-9dd2-aa61764b3407 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:45.391385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541058c5-1097-412d-9dd2-aa61764b3407 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:45.391385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541058c5-1097-412d-9dd2-aa61764b3407 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:45.392799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 25/47] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:45.398577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b649958-7011-476e-8ceb-1c2aa3e08382 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:45.398951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b649958-7011-476e-8ceb-1c2aa3e08382 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:45.399136 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b649958-7011-476e-8ceb-1c2aa3e08382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:45.418425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-6b649958-7011-476e-8ceb-1c2aa3e08382 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:45.419976 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6b649958-7011-476e-8ceb-1c2aa3e08382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:45.449285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:45.490656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 23/48] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:45.757229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541058c5-1097-412d-9dd2-aa61764b3407 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:45.868655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541058c5-1097-412d-9dd2-aa61764b3407 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:45.883056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cdca5a-cb8a-412f-8b31-63d0ef6e1efa None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:45.883353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cdca5a-cb8a-412f-8b31-63d0ef6e1efa None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:45.883619 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cdca5a-cb8a-412f-8b31-63d0ef6e1efa None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:45.884989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 24/49] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:15:45.890235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9f403-beb1-4660-a4c6-08ce4df0629d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:45.890522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9f403-beb1-4660-a4c6-08ce4df0629d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:45.890798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9f403-beb1-4660-a4c6-08ce4df0629d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:45.905179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ad76c6-ccb3-4f63-abe6-55d9a23d05b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:45.906134 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad76c6-ccb3-4f63-abe6-55d9a23d05b7 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:45.906431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad76c6-ccb3-4f63-abe6-55d9a23d05b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:45.906739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ad76c6-ccb3-4f63-abe6-55d9a23d05b7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:45.908801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-5cc9f403-beb1-4660-a4c6-08ce4df0629d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:45.910031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5cc9f403-beb1-4660-a4c6-08ce4df0629d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:45.917247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ad76c6-ccb3-4f63-abe6-55d9a23d05b7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-56ad76c6-ccb3-4f63-abe6-55d9a23d05b7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:15:45.922144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:45.923105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:15:45.924369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 26/50] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:15:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:15:45.929794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ad76c6-ccb3-4f63-abe6-55d9a23d05b7 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:45.978668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:45.981938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 25/51] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:15:46.032346 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ad76c6-ccb3-4f63-abe6-55d9a23d05b7 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:46.066774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2f340a-d47d-4a3c-b0c5-6be8115733c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:46.067385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2f340a-d47d-4a3c-b0c5-6be8115733c4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:46.067649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2f340a-d47d-4a3c-b0c5-6be8115733c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:46.067923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2f340a-d47d-4a3c-b0c5-6be8115733c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:46.068597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2f340a-d47d-4a3c-b0c5-6be8115733c4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:46.068904 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2f340a-d47d-4a3c-b0c5-6be8115733c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:46.070493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2f340a-d47d-4a3c-b0c5-6be8115733c4 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:46.078515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 27/52] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:15:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:15:46.084551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9f403-beb1-4660-a4c6-08ce4df0629d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:46.179224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:46.187481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9f403-beb1-4660-a4c6-08ce4df0629d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:46.222706 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b9873d-c724-4aba-9c70-c57bd372b460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:46.223347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b9873d-c724-4aba-9c70-c57bd372b460 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:46.223628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b9873d-c724-4aba-9c70-c57bd372b460 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:46.223904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b9873d-c724-4aba-9c70-c57bd372b460 None admin] PATH_INFO: `/v3/users/d29c3b9b96c24a7daf5c64be528badeb` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:46.225042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b9873d-c724-4aba-9c70-c57bd372b460 None admin] RBAC: Authorizing `identity:get_user(user_id=d29c3b9b96c24a7daf5c64be528badeb)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:46.226547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b9873d-c724-4aba-9c70-c57bd372b460 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:46.228133 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 26/53] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:15:46 2022] GET /identity/v3/users/d29c3b9b96c24a7daf5c64be528badeb => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:46.234331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2f340a-d47d-4a3c-b0c5-6be8115733c4 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:46.243287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b9873d-c724-4aba-9c70-c57bd372b460 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:46.338903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2f340a-d47d-4a3c-b0c5-6be8115733c4 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:46.351963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:46.380928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ade658b-a88d-4a30-8409-8299871689e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:46.381595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ade658b-a88d-4a30-8409-8299871689e6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:46.381880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ade658b-a88d-4a30-8409-8299871689e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:46.382296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ade658b-a88d-4a30-8409-8299871689e6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:46.383679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ade658b-a88d-4a30-8409-8299871689e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:46.385188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ade658b-a88d-4a30-8409-8299871689e6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:46.386873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 28/54] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:15:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:46.393339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ade658b-a88d-4a30-8409-8299871689e6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:46.455665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b9873d-c724-4aba-9c70-c57bd372b460 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:46.489598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a5c489-ff12-4dd1-bf75-bddeae804df2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:46.490026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a5c489-ff12-4dd1-bf75-bddeae804df2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:46.490336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a5c489-ff12-4dd1-bf75-bddeae804df2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:46.490606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a5c489-ff12-4dd1-bf75-bddeae804df2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:46.491736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a5c489-ff12-4dd1-bf75-bddeae804df2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:46.493196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a5c489-ff12-4dd1-bf75-bddeae804df2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:46.494811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 27/55] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:15:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:15:46.497810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ade658b-a88d-4a30-8409-8299871689e6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:46.500704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a5c489-ff12-4dd1-bf75-bddeae804df2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:46.531309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:46.536721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea99ff5-376d-4e91-99b8-2a80f00c8281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:46.537333 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea99ff5-376d-4e91-99b8-2a80f00c8281 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:46.537614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea99ff5-376d-4e91-99b8-2a80f00c8281 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:46.537892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea99ff5-376d-4e91-99b8-2a80f00c8281 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:46.538668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea99ff5-376d-4e91-99b8-2a80f00c8281 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.domain.id=default, user.id=d29c3b9b96c24a7daf5c64be528badeb) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:46.538977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea99ff5-376d-4e91-99b8-2a80f00c8281 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:46.540453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea99ff5-376d-4e91-99b8-2a80f00c8281 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:46.546311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 29/56] 10.222.0.16 () {56 vars in 1322 bytes} [Tue Sep 13 15:15:46 2022] GET /identity/v3/role_assignments?user.id=d29c3b9b96c24a7daf5c64be528badeb&scope.domain.id=default&role.id=22819414415942d399db534708fe7103 => generated 530 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:46.603418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a5c489-ff12-4dd1-bf75-bddeae804df2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:46.642460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537d7e94-02ba-4466-8a87-3ee52137eee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:46.643364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537d7e94-02ba-4466-8a87-3ee52137eee0 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:46.643787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537d7e94-02ba-4466-8a87-3ee52137eee0 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:46.644809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537d7e94-02ba-4466-8a87-3ee52137eee0 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:46.644959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537d7e94-02ba-4466-8a87-3ee52137eee0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:46.646455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537d7e94-02ba-4466-8a87-3ee52137eee0 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:46.677527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 28/57] 10.222.0.16 () {60 vars in 1140 bytes} [Tue Sep 13 15:15:46 2022] POST /identity/v3/projects => generated 298 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:15:48.805080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff6228a-05a6-4678-b1df-0ffc45048592 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:48.805626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff6228a-05a6-4678-b1df-0ffc45048592 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:48.805626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff6228a-05a6-4678-b1df-0ffc45048592 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:48.807018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 30/58] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:48.813529 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dfd269-723a-4635-99dd-d60648a81854 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:48.814250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dfd269-723a-4635-99dd-d60648a81854 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:48.814250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dfd269-723a-4635-99dd-d60648a81854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:48.843694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b7dfd269-723a-4635-99dd-d60648a81854 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:48.844903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b7dfd269-723a-4635-99dd-d60648a81854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:48.912722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 29/59] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:49.187365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff6228a-05a6-4678-b1df-0ffc45048592 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:49.236866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:49.300252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff6228a-05a6-4678-b1df-0ffc45048592 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:49.338846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321902f6-4d5d-4642-9bc5-29b4c1079079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:49.339478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321902f6-4d5d-4642-9bc5-29b4c1079079 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:49.339764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321902f6-4d5d-4642-9bc5-29b4c1079079 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:49.340042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321902f6-4d5d-4642-9bc5-29b4c1079079 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:49.341226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321902f6-4d5d-4642-9bc5-29b4c1079079 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:49.342727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321902f6-4d5d-4642-9bc5-29b4c1079079 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:49.344514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 31/60] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:15:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:49.351108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dfd269-723a-4635-99dd-d60648a81854 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:49.374764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:49.473030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dfd269-723a-4635-99dd-d60648a81854 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:49.508456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc63d37-fe8d-419f-a7a5-d94391502ced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:49.509058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc63d37-fe8d-419f-a7a5-d94391502ced None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:49.509329 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc63d37-fe8d-419f-a7a5-d94391502ced None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:49.509600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc63d37-fe8d-419f-a7a5-d94391502ced None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:49.510274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc63d37-fe8d-419f-a7a5-d94391502ced None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:49.511708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc63d37-fe8d-419f-a7a5-d94391502ced None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:49.539415 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 30/61] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:15:49 2022] POST /identity/v3/users => generated 275 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:15:52.709879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12df29d-e415-4ede-bbbb-e11d41b30f46 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:52.710943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12df29d-e415-4ede-bbbb-e11d41b30f46 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:52.711433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12df29d-e415-4ede-bbbb-e11d41b30f46 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:52.712975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 32/62] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:52.718645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e98dfd-171e-4227-8adf-2db7d0130fc5 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:52.719577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e98dfd-171e-4227-8adf-2db7d0130fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:52.720105 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e98dfd-171e-4227-8adf-2db7d0130fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:52.745597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-75e98dfd-171e-4227-8adf-2db7d0130fc5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:52.747220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-75e98dfd-171e-4227-8adf-2db7d0130fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:52.838526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:52.846641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 31/63] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:52.935531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa24cfe-4c98-4ef6-a989-78ad0da2fc55 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:52.935847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa24cfe-4c98-4ef6-a989-78ad0da2fc55 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:52.936090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa24cfe-4c98-4ef6-a989-78ad0da2fc55 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:52.937632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 33/64] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:52.942803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ff83d-10d2-4749-b38d-0538859d3ffc None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:52.943091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ff83d-10d2-4749-b38d-0538859d3ffc None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:52.943366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ff83d-10d2-4749-b38d-0538859d3ffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:52.967155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-740ff83d-10d2-4749-b38d-0538859d3ffc None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:52.968523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-740ff83d-10d2-4749-b38d-0538859d3ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:53.060514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 32/65] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:53.115553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9f27ab-7154-4005-bfc1-9421eef754e6 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.116173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9f27ab-7154-4005-bfc1-9421eef754e6 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.116668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9f27ab-7154-4005-bfc1-9421eef754e6 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.118272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 34/66] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:53.123542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736603fa-81ca-4746-a1eb-264a25aa7bc8 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.123826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736603fa-81ca-4746-a1eb-264a25aa7bc8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.124148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736603fa-81ca-4746-a1eb-264a25aa7bc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.147078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-736603fa-81ca-4746-a1eb-264a25aa7bc8 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:53.148262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-736603fa-81ca-4746-a1eb-264a25aa7bc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:53.204970 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9f27ab-7154-4005-bfc1-9421eef754e6 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:53.236509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 33/67] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:53.367960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9f27ab-7154-4005-bfc1-9421eef754e6 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:53.375529 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736603fa-81ca-4746-a1eb-264a25aa7bc8 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:53.382460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:53.423977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ce5150-88b0-4bd2-aab4-7a4bbf05a5db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:53.425195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ce5150-88b0-4bd2-aab4-7a4bbf05a5db None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.426364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ce5150-88b0-4bd2-aab4-7a4bbf05a5db None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.427000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ce5150-88b0-4bd2-aab4-7a4bbf05a5db None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.451520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ce5150-88b0-4bd2-aab4-7a4bbf05a5db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-68ce5150-88b0-4bd2-aab4-7a4bbf05a5db None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:15:53.457186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:53.458018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:15:53.458809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:15:53.462919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 35/68] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:15:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:15:53.470932 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ce5150-88b0-4bd2-aab4-7a4bbf05a5db None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:53.487179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:53.542885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736603fa-81ca-4746-a1eb-264a25aa7bc8 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:53.581393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ce5150-88b0-4bd2-aab4-7a4bbf05a5db None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:53.586138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e1f7c1-061e-4c81-b5ff-925883adf747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:53.586924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1f7c1-061e-4c81-b5ff-925883adf747 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.587304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1f7c1-061e-4c81-b5ff-925883adf747 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.587676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e1f7c1-061e-4c81-b5ff-925883adf747 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.600793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e1f7c1-061e-4c81-b5ff-925883adf747 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-c1e1f7c1-061e-4c81-b5ff-925883adf747 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:15:53.607221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:53.608080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:15:53.608905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:15:53.612480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 34/69] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:15:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:15:53.615096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55e8114-7f4a-45c4-a6ed-ed3e1dc52b8b None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.615604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55e8114-7f4a-45c4-a6ed-ed3e1dc52b8b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.616071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55e8114-7f4a-45c4-a6ed-ed3e1dc52b8b None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.617661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 35/70] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:53.618599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55e8114-7f4a-45c4-a6ed-ed3e1dc52b8b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:53.624898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86335eb3-476f-4873-9a7d-2419d3784a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:53.625529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86335eb3-476f-4873-9a7d-2419d3784a83 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.625807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86335eb3-476f-4873-9a7d-2419d3784a83 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.626076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86335eb3-476f-4873-9a7d-2419d3784a83 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.626778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86335eb3-476f-4873-9a7d-2419d3784a83 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:53.627080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86335eb3-476f-4873-9a7d-2419d3784a83 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:53.628562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86335eb3-476f-4873-9a7d-2419d3784a83 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:53.637435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 36/71] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:15:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:53.643732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ffcf5b-2428-42ac-9257-dea7fe60d2ce None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.644219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ffcf5b-2428-42ac-9257-dea7fe60d2ce None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.644659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ffcf5b-2428-42ac-9257-dea7fe60d2ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.663243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-a2ffcf5b-2428-42ac-9257-dea7fe60d2ce None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:53.664753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a2ffcf5b-2428-42ac-9257-dea7fe60d2ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:53.698062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:53.739479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55e8114-7f4a-45c4-a6ed-ed3e1dc52b8b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:53.745695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 37/72] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:53.746581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ffcf5b-2428-42ac-9257-dea7fe60d2ce None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:53.766066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:53.791770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8be584-5484-4acd-b6e0-eee5d348fdab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:53.792406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8be584-5484-4acd-b6e0-eee5d348fdab None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.792676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8be584-5484-4acd-b6e0-eee5d348fdab None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.792948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8be584-5484-4acd-b6e0-eee5d348fdab None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.793649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8be584-5484-4acd-b6e0-eee5d348fdab None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:53.793929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8be584-5484-4acd-b6e0-eee5d348fdab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:53.795389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8be584-5484-4acd-b6e0-eee5d348fdab None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:53.805578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 36/73] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:15:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:53.806946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8be584-5484-4acd-b6e0-eee5d348fdab None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:53.867052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ffcf5b-2428-42ac-9257-dea7fe60d2ce None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:53.901955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d622532-47aa-4448-97d2-5d8028677cc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:53.902632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d622532-47aa-4448-97d2-5d8028677cc2 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.902934 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d622532-47aa-4448-97d2-5d8028677cc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.903209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d622532-47aa-4448-97d2-5d8028677cc2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.912115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8be584-5484-4acd-b6e0-eee5d348fdab None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:53.913433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d622532-47aa-4448-97d2-5d8028677cc2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-3d622532-47aa-4448-97d2-5d8028677cc2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:15:53.918266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:53.919283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:15:53.920221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:15:53.922142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 38/74] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:15:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:15:53.923065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d622532-47aa-4448-97d2-5d8028677cc2 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:53.947753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:53.953731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4ba7bf-6efb-40e4-9542-e12e061a1b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:53.954439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4ba7bf-6efb-40e4-9542-e12e061a1b4b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.954719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4ba7bf-6efb-40e4-9542-e12e061a1b4b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.954992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4ba7bf-6efb-40e4-9542-e12e061a1b4b None admin] PATH_INFO: `/v3/users/d29c3b9b96c24a7daf5c64be528badeb` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.956182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4ba7bf-6efb-40e4-9542-e12e061a1b4b None admin] RBAC: Authorizing `identity:get_user(user_id=d29c3b9b96c24a7daf5c64be528badeb)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:53.957635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4ba7bf-6efb-40e4-9542-e12e061a1b4b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:53.959259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 37/75] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:15:53 2022] GET /identity/v3/users/d29c3b9b96c24a7daf5c64be528badeb => generated 247 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:53.964841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d6673c-02fe-4bd8-ac22-372f2002a3f5 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:53.965181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d6673c-02fe-4bd8-ac22-372f2002a3f5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:53.965464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d6673c-02fe-4bd8-ac22-372f2002a3f5 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:53.966727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 38/76] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:53 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:53.971343 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d6673c-02fe-4bd8-ac22-372f2002a3f5 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:54.023431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:54.033031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d622532-47aa-4448-97d2-5d8028677cc2 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:54.068810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ec358a-7eb5-4fd6-838b-028b51d1804d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:54.070032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec358a-7eb5-4fd6-838b-028b51d1804d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.070526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec358a-7eb5-4fd6-838b-028b51d1804d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.070995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec358a-7eb5-4fd6-838b-028b51d1804d None admin] PATH_INFO: `/v3/users/a734913c72ea45f3abe03c5a9b7f9695` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.075953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d6673c-02fe-4bd8-ac22-372f2002a3f5 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:54.087660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ec358a-7eb5-4fd6-838b-028b51d1804d None admin] RBAC: Authorizing `identity:get_user(user_id=a734913c72ea45f3abe03c5a9b7f9695)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:54.089171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ec358a-7eb5-4fd6-838b-028b51d1804d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:54.090921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 39/77] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:15:53 2022] GET /identity/v3/users/a734913c72ea45f3abe03c5a9b7f9695 => generated 275 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:54.091694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ec358a-7eb5-4fd6-838b-028b51d1804d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:54.111678 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f26593f-b9b1-4c81-8d06-4e9d8f578103 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:54.112590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f26593f-b9b1-4c81-8d06-4e9d8f578103 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.113346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f26593f-b9b1-4c81-8d06-4e9d8f578103 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.113768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f26593f-b9b1-4c81-8d06-4e9d8f578103 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.114829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f26593f-b9b1-4c81-8d06-4e9d8f578103 None admin] RBAC: Adding query filter params (name=member) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:54.115300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f26593f-b9b1-4c81-8d06-4e9d8f578103 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:54.117459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f26593f-b9b1-4c81-8d06-4e9d8f578103 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:54.127213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 39/78] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:15:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:54.128659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4c8ad0-4ae2-4424-8679-c91c001b2555 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.128947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4c8ad0-4ae2-4424-8679-c91c001b2555 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.129220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4c8ad0-4ae2-4424-8679-c91c001b2555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.147967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-5b4c8ad0-4ae2-4424-8679-c91c001b2555 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:54.149186 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5b4c8ad0-4ae2-4424-8679-c91c001b2555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:54.178067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:54.197908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ec358a-7eb5-4fd6-838b-028b51d1804d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:54.221077 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 40/79] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:54.221926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4c8ad0-4ae2-4424-8679-c91c001b2555 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:54.248155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b71f977-6c66-4498-9202-b5af09c5b744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:54.248860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b71f977-6c66-4498-9202-b5af09c5b744 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.249182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b71f977-6c66-4498-9202-b5af09c5b744 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.249455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b71f977-6c66-4498-9202-b5af09c5b744 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.263643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:54.330398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4c8ad0-4ae2-4424-8679-c91c001b2555 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:54.330561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b71f977-6c66-4498-9202-b5af09c5b744 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:54.332249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b71f977-6c66-4498-9202-b5af09c5b744 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:54.365235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07de05b-fb22-4be5-af84-6cf8da62c728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:54.366037 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07de05b-fb22-4be5-af84-6cf8da62c728 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.366357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07de05b-fb22-4be5-af84-6cf8da62c728 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.366955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07de05b-fb22-4be5-af84-6cf8da62c728 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.378638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07de05b-fb22-4be5-af84-6cf8da62c728 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-c07de05b-fb22-4be5-af84-6cf8da62c728 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:15:54.384187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:54.385217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 41/80] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:15:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:15:54.386857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07de05b-fb22-4be5-af84-6cf8da62c728 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:54.428832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:54.457209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 40/81] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:15:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:15:54.457209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b71f977-6c66-4498-9202-b5af09c5b744 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:54.498710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:54.501984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07de05b-fb22-4be5-af84-6cf8da62c728 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:54.536129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e712653f-e888-45f3-91d7-208483bc33ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:54.536744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e712653f-e888-45f3-91d7-208483bc33ae None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.537024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e712653f-e888-45f3-91d7-208483bc33ae None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.537297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e712653f-e888-45f3-91d7-208483bc33ae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.556667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b71f977-6c66-4498-9202-b5af09c5b744 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:54.590364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95836f33-f946-4d8a-869b-06caaec4ad7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:54.590930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95836f33-f946-4d8a-869b-06caaec4ad7e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.591208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95836f33-f946-4d8a-869b-06caaec4ad7e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.591478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95836f33-f946-4d8a-869b-06caaec4ad7e None admin] PATH_INFO: `/v3/users/a734913c72ea45f3abe03c5a9b7f9695` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.592601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95836f33-f946-4d8a-869b-06caaec4ad7e None admin] RBAC: Authorizing `identity:get_user(user_id=a734913c72ea45f3abe03c5a9b7f9695)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:54.594245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95836f33-f946-4d8a-869b-06caaec4ad7e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:54.595856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 41/82] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:15:54 2022] GET /identity/v3/users/a734913c72ea45f3abe03c5a9b7f9695 => generated 275 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:54.596648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95836f33-f946-4d8a-869b-06caaec4ad7e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:54.606961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e712653f-e888-45f3-91d7-208483bc33ae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:54.608458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e712653f-e888-45f3-91d7-208483bc33ae None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:54.654455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:54.702906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:54.704918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95836f33-f946-4d8a-869b-06caaec4ad7e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:54.711870 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 42/83] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:15:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:15:54.712458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e712653f-e888-45f3-91d7-208483bc33ae None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:54.737942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd7a277-7233-4d24-b22d-345dac8a4e0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:54.738928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd7a277-7233-4d24-b22d-345dac8a4e0d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.739203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd7a277-7233-4d24-b22d-345dac8a4e0d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.739483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd7a277-7233-4d24-b22d-345dac8a4e0d None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.740114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd7a277-7233-4d24-b22d-345dac8a4e0d None admin] RBAC: Adding query filter params (name=member) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:54.740413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd7a277-7233-4d24-b22d-345dac8a4e0d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:54.741809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd7a277-7233-4d24-b22d-345dac8a4e0d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:54.749662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 42/84] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:15:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:54.750515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd7a277-7233-4d24-b22d-345dac8a4e0d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:54.811402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e712653f-e888-45f3-91d7-208483bc33ae None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:54.844691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c783809-9840-497f-b034-98353b66fd17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:54.845242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c783809-9840-497f-b034-98353b66fd17 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.845523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c783809-9840-497f-b034-98353b66fd17 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.845797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c783809-9840-497f-b034-98353b66fd17 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.846168 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd7a277-7233-4d24-b22d-345dac8a4e0d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:54.846952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c783809-9840-497f-b034-98353b66fd17 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:54.848480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c783809-9840-497f-b034-98353b66fd17 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:54.850110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 43/85] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:15:54 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:54.850900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c783809-9840-497f-b034-98353b66fd17 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:54.858649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:54.883623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03344c4-7f80-4976-aa8c-b59e539d818c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:54.884256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03344c4-7f80-4976-aa8c-b59e539d818c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.884533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03344c4-7f80-4976-aa8c-b59e539d818c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.884806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03344c4-7f80-4976-aa8c-b59e539d818c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.885932 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03344c4-7f80-4976-aa8c-b59e539d818c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:54.887441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03344c4-7f80-4976-aa8c-b59e539d818c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:54.889017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 43/86] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:15:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:54.890893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03344c4-7f80-4976-aa8c-b59e539d818c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:54.916188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:54.950643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c783809-9840-497f-b034-98353b66fd17 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:54.984126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73100a3-61a4-46f5-83e7-7d374ffa8870 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:54.984754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73100a3-61a4-46f5-83e7-7d374ffa8870 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:54.985030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73100a3-61a4-46f5-83e7-7d374ffa8870 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:54.985303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73100a3-61a4-46f5-83e7-7d374ffa8870 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:54.989091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03344c4-7f80-4976-aa8c-b59e539d818c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:55.021646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8a532f-47a3-4dad-8384-dff92c2c0545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:55.022331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8a532f-47a3-4dad-8384-dff92c2c0545 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:55.022605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8a532f-47a3-4dad-8384-dff92c2c0545 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:55.022869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8a532f-47a3-4dad-8384-dff92c2c0545 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:55.023992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8a532f-47a3-4dad-8384-dff92c2c0545 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:55.025440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8a532f-47a3-4dad-8384-dff92c2c0545 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:55.027083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 44/87] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:15:54 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:55.027835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8a532f-47a3-4dad-8384-dff92c2c0545 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:55.053641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73100a3-61a4-46f5-83e7-7d374ffa8870 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:55.055187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73100a3-61a4-46f5-83e7-7d374ffa8870 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:55.072243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:55.120446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:55.125591 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8a532f-47a3-4dad-8384-dff92c2c0545 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:55.156286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 44/88] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:15:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:15:55.156918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73100a3-61a4-46f5-83e7-7d374ffa8870 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:55.158279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a063fbae-207a-41fb-8ae9-d182a6670acd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:55.158925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a063fbae-207a-41fb-8ae9-d182a6670acd None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:55.159209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a063fbae-207a-41fb-8ae9-d182a6670acd None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:55.159489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a063fbae-207a-41fb-8ae9-d182a6670acd None admin] PATH_INFO: `/v3/users/a734913c72ea45f3abe03c5a9b7f9695` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:55.160724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a063fbae-207a-41fb-8ae9-d182a6670acd None admin] RBAC: Authorizing `identity:get_user(user_id=a734913c72ea45f3abe03c5a9b7f9695)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:55.162243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a063fbae-207a-41fb-8ae9-d182a6670acd None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:55.163827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 45/89] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:15:55 2022] GET /identity/v3/users/a734913c72ea45f3abe03c5a9b7f9695 => generated 275 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:55.166943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a063fbae-207a-41fb-8ae9-d182a6670acd None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:55.262962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73100a3-61a4-46f5-83e7-7d374ffa8870 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:55.265680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a063fbae-207a-41fb-8ae9-d182a6670acd None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:55.279124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:55.297124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ead678a-4317-415c-b17e-cce2f02c36d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:55.297742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ead678a-4317-415c-b17e-cce2f02c36d0 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:55.298018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ead678a-4317-415c-b17e-cce2f02c36d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:55.298316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ead678a-4317-415c-b17e-cce2f02c36d0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:55.299044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ead678a-4317-415c-b17e-cce2f02c36d0 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517, user.id=d29c3b9b96c24a7daf5c64be528badeb) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:55.299291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a71f31c-dd72-4e6e-a309-9d2833069769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:55.299393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ead678a-4317-415c-b17e-cce2f02c36d0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:55.299910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a71f31c-dd72-4e6e-a309-9d2833069769 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:55.300189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a71f31c-dd72-4e6e-a309-9d2833069769 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:55.300460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a71f31c-dd72-4e6e-a309-9d2833069769 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:55.300774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ead678a-4317-415c-b17e-cce2f02c36d0 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:55.301461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a71f31c-dd72-4e6e-a309-9d2833069769 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:55.302948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a71f31c-dd72-4e6e-a309-9d2833069769 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:55.308131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 45/90] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:15:55 2022] GET /identity/v3/role_assignments?user.id=d29c3b9b96c24a7daf5c64be528badeb&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=22819414415942d399db534708fe7103 => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:55.308906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ead678a-4317-415c-b17e-cce2f02c36d0 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:55.335316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 46/91] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:15:55 2022] POST /identity/v3/users => generated 289 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:15:55.335990 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a71f31c-dd72-4e6e-a309-9d2833069769 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:55.368926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:55.464993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ead678a-4317-415c-b17e-cce2f02c36d0 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:55.475629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a71f31c-dd72-4e6e-a309-9d2833069769 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:55.505643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd2853-7d52-457b-9a1b-ef3626cb95c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:55.507118 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd2853-7d52-457b-9a1b-ef3626cb95c9 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:55.507118 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd2853-7d52-457b-9a1b-ef3626cb95c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:55.507118 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd2853-7d52-457b-9a1b-ef3626cb95c9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:55.507995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fd2853-7d52-457b-9a1b-ef3626cb95c9 None admin] RBAC: Adding query filter params (role.id=74db8755b1f04d839260ccd3e36deb66, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517, user.id=a734913c72ea45f3abe03c5a9b7f9695) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:55.508255 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fd2853-7d52-457b-9a1b-ef3626cb95c9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:55.509932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fd2853-7d52-457b-9a1b-ef3626cb95c9 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:55.516263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 46/92] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:15:55 2022] GET /identity/v3/role_assignments?user.id=a734913c72ea45f3abe03c5a9b7f9695&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=74db8755b1f04d839260ccd3e36deb66 => generated 259 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:55.517235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd2853-7d52-457b-9a1b-ef3626cb95c9 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:55.521741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab86b8b2-04f8-47db-b874-353b0533ec3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:55.522453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab86b8b2-04f8-47db-b874-353b0533ec3e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:55.522735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab86b8b2-04f8-47db-b874-353b0533ec3e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:55.523012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab86b8b2-04f8-47db-b874-353b0533ec3e None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:55.524277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab86b8b2-04f8-47db-b874-353b0533ec3e None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:55.525984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab86b8b2-04f8-47db-b874-353b0533ec3e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:55.528074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 47/93] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:15:55 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:55.536566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab86b8b2-04f8-47db-b874-353b0533ec3e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:55.601999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:55.647726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:55.656428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd2853-7d52-457b-9a1b-ef3626cb95c9 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:55.711984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b6909-6686-4a7a-a0fa-5a043738ff44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:55.712692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b6909-6686-4a7a-a0fa-5a043738ff44 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:55.712977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b6909-6686-4a7a-a0fa-5a043738ff44 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:55.713248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b6909-6686-4a7a-a0fa-5a043738ff44 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:55.725393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab86b8b2-04f8-47db-b874-353b0533ec3e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:55.784535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200dd963-e177-4f8d-8efc-0e2f5e111e96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:55.785159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200dd963-e177-4f8d-8efc-0e2f5e111e96 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:55.785159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200dd963-e177-4f8d-8efc-0e2f5e111e96 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:55.785380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200dd963-e177-4f8d-8efc-0e2f5e111e96 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:55.786149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200dd963-e177-4f8d-8efc-0e2f5e111e96 None admin] RBAC: Adding query filter params (role.id=7a7e816668a7493bad0752baca37934c, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517, user.id=a734913c72ea45f3abe03c5a9b7f9695) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:55.786472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200dd963-e177-4f8d-8efc-0e2f5e111e96 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:55.788844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200dd963-e177-4f8d-8efc-0e2f5e111e96 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:55.795174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 48/94] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:15:55 2022] GET /identity/v3/role_assignments?user.id=a734913c72ea45f3abe03c5a9b7f9695&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=7a7e816668a7493bad0752baca37934c => generated 259 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:15:55.822410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b6909-6686-4a7a-a0fa-5a043738ff44 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:55.824005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b6909-6686-4a7a-a0fa-5a043738ff44 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:55.931516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:55.938775 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 47/95] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:15:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:15:55.945082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200dd963-e177-4f8d-8efc-0e2f5e111e96 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:56.061532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200dd963-e177-4f8d-8efc-0e2f5e111e96 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:56.063085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:56.101139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bdf3ff-fdf5-46d7-b5b8-41609424f4b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:56.101798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bdf3ff-fdf5-46d7-b5b8-41609424f4b5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:56.102083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bdf3ff-fdf5-46d7-b5b8-41609424f4b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:56.102390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bdf3ff-fdf5-46d7-b5b8-41609424f4b5 None admin] PATH_INFO: `/v3/projects/e38dc6d96ea24fb296893d8dfb010287` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:56.103530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bdf3ff-fdf5-46d7-b5b8-41609424f4b5 None admin] RBAC: Authorizing `identity:get_project(project_id=e38dc6d96ea24fb296893d8dfb010287)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:56.105128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bdf3ff-fdf5-46d7-b5b8-41609424f4b5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:56.107003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 49/96] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:15:55 2022] GET /identity/v3/projects/e38dc6d96ea24fb296893d8dfb010287 => generated 312 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:56.113470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b6909-6686-4a7a-a0fa-5a043738ff44 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:56.240110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b6909-6686-4a7a-a0fa-5a043738ff44 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:56.282757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff2c77a-c678-4edb-9fae-05d9048b7315 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:56.283532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff2c77a-c678-4edb-9fae-05d9048b7315 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:56.283816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff2c77a-c678-4edb-9fae-05d9048b7315 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:56.284097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff2c77a-c678-4edb-9fae-05d9048b7315 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:56.284845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff2c77a-c678-4edb-9fae-05d9048b7315 None admin] RBAC: Adding query filter params (role.id=7a7e816668a7493bad0752baca37934c, scope.project.id=e38dc6d96ea24fb296893d8dfb010287, user.id=a734913c72ea45f3abe03c5a9b7f9695) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:56.285158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff2c77a-c678-4edb-9fae-05d9048b7315 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:56.286653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff2c77a-c678-4edb-9fae-05d9048b7315 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:56.297921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 48/97] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:15:56 2022] GET /identity/v3/role_assignments?user.id=a734913c72ea45f3abe03c5a9b7f9695&scope.project.id=e38dc6d96ea24fb296893d8dfb010287&role.id=7a7e816668a7493bad0752baca37934c => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:58.288287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c42d80b-3b6b-4fed-a1a9-c2451c6ca44d None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:58.289300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c42d80b-3b6b-4fed-a1a9-c2451c6ca44d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:58.289778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c42d80b-3b6b-4fed-a1a9-c2451c6ca44d None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:58.291372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 50/98] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:58.297472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24685e3d-546b-432c-8fc0-8b103144335f None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:58.298355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24685e3d-546b-432c-8fc0-8b103144335f None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:58.298828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24685e3d-546b-432c-8fc0-8b103144335f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:58.323165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-24685e3d-546b-432c-8fc0-8b103144335f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:58.324711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-24685e3d-546b-432c-8fc0-8b103144335f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:58.395349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:58.411644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 49/99] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:58.573229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98da73e4-a740-4539-841d-81443f29bd2e None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:58.573837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98da73e4-a740-4539-841d-81443f29bd2e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:58.574405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98da73e4-a740-4539-841d-81443f29bd2e None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:58.576215 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 51/100] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:58.582394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3e3c8f-4141-4c2b-a288-ae4df333055d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:58.582937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3e3c8f-4141-4c2b-a288-ae4df333055d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:58.583418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3e3c8f-4141-4c2b-a288-ae4df333055d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:58.602488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8e3e3c8f-4141-4c2b-a288-ae4df333055d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:58.604163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8e3e3c8f-4141-4c2b-a288-ae4df333055d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:58.687879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 50/101] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:58.825501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cd6fe3-6910-4d8a-be8f-4e8855395c52 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:58.826206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cd6fe3-6910-4d8a-be8f-4e8855395c52 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:58.826693 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cd6fe3-6910-4d8a-be8f-4e8855395c52 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:58.828297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 52/102] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:15:58.833509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512dac24-c431-40df-bcbf-fde83e96c307 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:58.834062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512dac24-c431-40df-bcbf-fde83e96c307 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:58.834549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512dac24-c431-40df-bcbf-fde83e96c307 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:58.859197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-512dac24-c431-40df-bcbf-fde83e96c307 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:58.860738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-512dac24-c431-40df-bcbf-fde83e96c307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:58.886378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cd6fe3-6910-4d8a-be8f-4e8855395c52 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:58.998968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 51/103] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:59.030893 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b626681-f6db-43a9-bb7a-9baa494907f5 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.031567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b626681-f6db-43a9-bb7a-9baa494907f5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.032037 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b626681-f6db-43a9-bb7a-9baa494907f5 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.033882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 52/104] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:15:59.039195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3660e8a8-f0aa-489a-8aa9-0f6e72489659 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.039690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3660e8a8-f0aa-489a-8aa9-0f6e72489659 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.040173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3660e8a8-f0aa-489a-8aa9-0f6e72489659 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.064016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-3660e8a8-f0aa-489a-8aa9-0f6e72489659 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:59.065648 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3660e8a8-f0aa-489a-8aa9-0f6e72489659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:59.070003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cd6fe3-6910-4d8a-be8f-4e8855395c52 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:59.131243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:59.144445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aed4226-f6dd-45cd-a1b2-f460cb858125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:59.145112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aed4226-f6dd-45cd-a1b2-f460cb858125 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.145385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aed4226-f6dd-45cd-a1b2-f460cb858125 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.145828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aed4226-f6dd-45cd-a1b2-f460cb858125 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.156313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 53/105] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:59.175028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aed4226-f6dd-45cd-a1b2-f460cb858125 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:15:59.178280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e21ba0-a398-47d7-a5e1-6281a74120e9 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.178777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e21ba0-a398-47d7-a5e1-6281a74120e9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.179270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e21ba0-a398-47d7-a5e1-6281a74120e9 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-4aed4226-f6dd-45cd-a1b2-f460cb858125 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:15:59.179877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:59.180683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:15:59.181795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 53/106] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:15:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:15:59.183843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 54/107] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:15:59 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:15:59.187641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aed4226-f6dd-45cd-a1b2-f460cb858125 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:59.189428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb7636f-e549-4800-8088-7dc1e1cb2014 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.189759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb7636f-e549-4800-8088-7dc1e1cb2014 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.190043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb7636f-e549-4800-8088-7dc1e1cb2014 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.218404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-feb7636f-e549-4800-8088-7dc1e1cb2014 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:15:59.219912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-feb7636f-e549-4800-8088-7dc1e1cb2014 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:15:59.258997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:59.335465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 55/108] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:15:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:15:59.337151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb7636f-e549-4800-8088-7dc1e1cb2014 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:59.441607 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aed4226-f6dd-45cd-a1b2-f460cb858125 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:59.504603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb7636f-e549-4800-8088-7dc1e1cb2014 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:59.508192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b75391-e267-43bc-baa5-804771bc9abd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:59.508834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b75391-e267-43bc-baa5-804771bc9abd None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.509116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b75391-e267-43bc-baa5-804771bc9abd None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.509388 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b75391-e267-43bc-baa5-804771bc9abd None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.510062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b75391-e267-43bc-baa5-804771bc9abd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:59.510383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b75391-e267-43bc-baa5-804771bc9abd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:59.511791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b75391-e267-43bc-baa5-804771bc9abd None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:59.519378 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:59.524696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 54/109] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:15:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:59.526361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b75391-e267-43bc-baa5-804771bc9abd None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:59.548635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663633ae-efd7-4fb9-8477-241926c9cd6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:59.549530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663633ae-efd7-4fb9-8477-241926c9cd6d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.549820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663633ae-efd7-4fb9-8477-241926c9cd6d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.550098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663633ae-efd7-4fb9-8477-241926c9cd6d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.562252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663633ae-efd7-4fb9-8477-241926c9cd6d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-663633ae-efd7-4fb9-8477-241926c9cd6d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:15:59.568181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:59.569052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:15:59.569830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 56/110] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:15:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:15:59.574864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663633ae-efd7-4fb9-8477-241926c9cd6d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:59.660004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:59.673015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b75391-e267-43bc-baa5-804771bc9abd None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:59.710026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663633ae-efd7-4fb9-8477-241926c9cd6d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:59.710925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540bdd54-98e5-4a91-a391-72809c6df15f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:59.711766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540bdd54-98e5-4a91-a391-72809c6df15f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.712052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540bdd54-98e5-4a91-a391-72809c6df15f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.712331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540bdd54-98e5-4a91-a391-72809c6df15f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.713536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540bdd54-98e5-4a91-a391-72809c6df15f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:59.715087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540bdd54-98e5-4a91-a391-72809c6df15f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:59.718222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 55/111] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:15:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:59.718222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540bdd54-98e5-4a91-a391-72809c6df15f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:59.754157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6210abaf-898c-4420-9f2b-5771b5c19003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:59.754915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6210abaf-898c-4420-9f2b-5771b5c19003 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.755197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6210abaf-898c-4420-9f2b-5771b5c19003 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.755479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6210abaf-898c-4420-9f2b-5771b5c19003 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.766937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6210abaf-898c-4420-9f2b-5771b5c19003 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-6210abaf-898c-4420-9f2b-5771b5c19003 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:15:59.772624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:15:59.773509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:15:59.774305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:15:59.775399 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 57/112] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:15:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:15:59.776237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6210abaf-898c-4420-9f2b-5771b5c19003 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:59.811997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:59.825855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540bdd54-98e5-4a91-a391-72809c6df15f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:59.857776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7958bb34-caae-4cbd-91be-1d5fddc56ced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:59.858440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7958bb34-caae-4cbd-91be-1d5fddc56ced None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.858815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7958bb34-caae-4cbd-91be-1d5fddc56ced None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.859093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7958bb34-caae-4cbd-91be-1d5fddc56ced None admin] PATH_INFO: `/v3/users/d29c3b9b96c24a7daf5c64be528badeb` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.860202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7958bb34-caae-4cbd-91be-1d5fddc56ced None admin] RBAC: Authorizing `identity:get_user(user_id=d29c3b9b96c24a7daf5c64be528badeb)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:59.861668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7958bb34-caae-4cbd-91be-1d5fddc56ced None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:59.863903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 56/113] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:15:59 2022] GET /identity/v3/users/d29c3b9b96c24a7daf5c64be528badeb => generated 247 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:59.864666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7958bb34-caae-4cbd-91be-1d5fddc56ced None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:59.884882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6210abaf-898c-4420-9f2b-5771b5c19003 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:59.891205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:15:59.918352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2de541-cd5a-442a-b154-7b498fbb8a54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:59.918934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2de541-cd5a-442a-b154-7b498fbb8a54 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.919211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2de541-cd5a-442a-b154-7b498fbb8a54 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.919477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2de541-cd5a-442a-b154-7b498fbb8a54 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:15:59.920145 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2de541-cd5a-442a-b154-7b498fbb8a54 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:15:59.920434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2de541-cd5a-442a-b154-7b498fbb8a54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:15:59.921855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2de541-cd5a-442a-b154-7b498fbb8a54 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:15:59.929770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 58/114] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:15:59 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:15:59.930454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2de541-cd5a-442a-b154-7b498fbb8a54 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:15:59.965731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7958bb34-caae-4cbd-91be-1d5fddc56ced None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:15:59.997942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01347c83-717a-4f99-91ed-e99e342b69d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:15:59.998609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01347c83-717a-4f99-91ed-e99e342b69d7 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:15:59.998887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01347c83-717a-4f99-91ed-e99e342b69d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:15:59.999203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01347c83-717a-4f99-91ed-e99e342b69d7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.009170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01347c83-717a-4f99-91ed-e99e342b69d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-01347c83-717a-4f99-91ed-e99e342b69d7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:00.014358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:00.015311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:00.016122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 57/115] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:15:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:00.016744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01347c83-717a-4f99-91ed-e99e342b69d7 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:00.022009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:00.031322 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2de541-cd5a-442a-b154-7b498fbb8a54 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:00.064341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c40c6a-7de2-476f-8edd-693e9b3e7d29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:00.064952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c40c6a-7de2-476f-8edd-693e9b3e7d29 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.065238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c40c6a-7de2-476f-8edd-693e9b3e7d29 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.065511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c40c6a-7de2-476f-8edd-693e9b3e7d29 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.066203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c40c6a-7de2-476f-8edd-693e9b3e7d29 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:00.067656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c40c6a-7de2-476f-8edd-693e9b3e7d29 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:00.094410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 59/116] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:15:59 2022] POST /identity/v3/projects => generated 302 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:16:00.095792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fde7fd-3300-40fd-bb2d-27b509be7346 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.096071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fde7fd-3300-40fd-bb2d-27b509be7346 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.096441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fde7fd-3300-40fd-bb2d-27b509be7346 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.097580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 60/117] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:00.101912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fde7fd-3300-40fd-bb2d-27b509be7346 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:00.119718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:00.138599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01347c83-717a-4f99-91ed-e99e342b69d7 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:00.179494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1217e7-3552-4c26-9610-3c97ad85668b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:00.180515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1217e7-3552-4c26-9610-3c97ad85668b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.180799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1217e7-3552-4c26-9610-3c97ad85668b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.181085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1217e7-3552-4c26-9610-3c97ad85668b None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.181775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1217e7-3552-4c26-9610-3c97ad85668b None admin] RBAC: Adding query filter params (name=member) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:00.182082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1217e7-3552-4c26-9610-3c97ad85668b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:00.183566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1217e7-3552-4c26-9610-3c97ad85668b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:00.198262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 58/118] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:00.198262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1217e7-3552-4c26-9610-3c97ad85668b None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:00.225865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fde7fd-3300-40fd-bb2d-27b509be7346 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:00.264064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5eaff5-ff4a-4de7-83fd-cc00a7037504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:00.264872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5eaff5-ff4a-4de7-83fd-cc00a7037504 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.265129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5eaff5-ff4a-4de7-83fd-cc00a7037504 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.265426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5eaff5-ff4a-4de7-83fd-cc00a7037504 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.266748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5eaff5-ff4a-4de7-83fd-cc00a7037504 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:00.268259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5eaff5-ff4a-4de7-83fd-cc00a7037504 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:00.270232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 61/119] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:00 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:00.270972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5eaff5-ff4a-4de7-83fd-cc00a7037504 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:00.292987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:00.313776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1217e7-3552-4c26-9610-3c97ad85668b None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:00.350818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdec4370-c65c-438f-a206-5f909c193e94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:00.351469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdec4370-c65c-438f-a206-5f909c193e94 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.351863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdec4370-c65c-438f-a206-5f909c193e94 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.352139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdec4370-c65c-438f-a206-5f909c193e94 None admin] PATH_INFO: `/v3/users/85841b4c5e464905b7f6869d24a1e483` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.366608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdec4370-c65c-438f-a206-5f909c193e94 None admin] RBAC: Authorizing `identity:get_user(user_id=85841b4c5e464905b7f6869d24a1e483)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:00.368308 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdec4370-c65c-438f-a206-5f909c193e94 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:00.370258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 59/120] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:00 2022] GET /identity/v3/users/85841b4c5e464905b7f6869d24a1e483 => generated 289 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:00.371335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fe0a5b-052f-4700-a1ba-bddeaddd87ac None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.371564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fe0a5b-052f-4700-a1ba-bddeaddd87ac None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.371855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fe0a5b-052f-4700-a1ba-bddeaddd87ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.390914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-20fe0a5b-052f-4700-a1ba-bddeaddd87ac None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:00.391752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5eaff5-ff4a-4de7-83fd-cc00a7037504 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:00.392206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-20fe0a5b-052f-4700-a1ba-bddeaddd87ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:00.395245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:00.427498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21351ce6-b7c5-42b1-928f-2890ab5a328d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:00.428125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21351ce6-b7c5-42b1-928f-2890ab5a328d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.428400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21351ce6-b7c5-42b1-928f-2890ab5a328d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.428675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21351ce6-b7c5-42b1-928f-2890ab5a328d None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.429345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21351ce6-b7c5-42b1-928f-2890ab5a328d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:00.429679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21351ce6-b7c5-42b1-928f-2890ab5a328d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:00.431630 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21351ce6-b7c5-42b1-928f-2890ab5a328d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:00.440097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 62/121] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:00 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:00.440983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21351ce6-b7c5-42b1-928f-2890ab5a328d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:00.478221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 60/122] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:00.478221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fe0a5b-052f-4700-a1ba-bddeaddd87ac None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:00.543405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:00.578225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21351ce6-b7c5-42b1-928f-2890ab5a328d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:00.592919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fe0a5b-052f-4700-a1ba-bddeaddd87ac None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:00.617924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecdaf0d-bd0b-44db-bae4-d3f231edb181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:00.618767 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecdaf0d-bd0b-44db-bae4-d3f231edb181 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.619161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecdaf0d-bd0b-44db-bae4-d3f231edb181 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.619537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecdaf0d-bd0b-44db-bae4-d3f231edb181 None admin] PATH_INFO: `/v3/users/a734913c72ea45f3abe03c5a9b7f9695` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.620758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecdaf0d-bd0b-44db-bae4-d3f231edb181 None admin] RBAC: Authorizing `identity:get_user(user_id=a734913c72ea45f3abe03c5a9b7f9695)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:00.624336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecdaf0d-bd0b-44db-bae4-d3f231edb181 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:00.625804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 63/123] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:00 2022] GET /identity/v3/users/a734913c72ea45f3abe03c5a9b7f9695 => generated 275 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:00.626695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecdaf0d-bd0b-44db-bae4-d3f231edb181 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:00.651396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cf1e8c-4093-4c15-a18d-b5630154175d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:00.652357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cf1e8c-4093-4c15-a18d-b5630154175d None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.652640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cf1e8c-4093-4c15-a18d-b5630154175d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.652920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cf1e8c-4093-4c15-a18d-b5630154175d None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/users/d29c3b9b96c24a7daf5c64be528badeb/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.655524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cf1e8c-4093-4c15-a18d-b5630154175d None admin] RBAC: Authorizing `identity:create_grant(project_id=1cf34cdf97b64f59bf465ae56dd45517, user_id=d29c3b9b96c24a7daf5c64be528badeb, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:00.657103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cf1e8c-4093-4c15-a18d-b5630154175d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:00.672188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:00.679616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 61/124] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:16:00 2022] PUT /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/users/d29c3b9b96c24a7daf5c64be528badeb/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:00.681065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cf1e8c-4093-4c15-a18d-b5630154175d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:00.764457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecdaf0d-bd0b-44db-bae4-d3f231edb181 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:00.806295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cf1e8c-4093-4c15-a18d-b5630154175d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:00.822152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f39a704-b580-4d72-85e1-b8b92497fd45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:00.823456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f39a704-b580-4d72-85e1-b8b92497fd45 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.824165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f39a704-b580-4d72-85e1-b8b92497fd45 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.824598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f39a704-b580-4d72-85e1-b8b92497fd45 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.845828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:00.852837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6876043c-0294-48e7-a29a-efd545c912b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:00.853678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6876043c-0294-48e7-a29a-efd545c912b6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:00.854057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6876043c-0294-48e7-a29a-efd545c912b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:00.854471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6876043c-0294-48e7-a29a-efd545c912b6 None admin] PATH_INFO: `/v3/users/a734913c72ea45f3abe03c5a9b7f9695` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:00.855920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6876043c-0294-48e7-a29a-efd545c912b6 None admin] RBAC: Authorizing `identity:get_user(user_id=a734913c72ea45f3abe03c5a9b7f9695)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:00.857517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6876043c-0294-48e7-a29a-efd545c912b6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:00.859489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 62/125] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:00 2022] GET /identity/v3/users/a734913c72ea45f3abe03c5a9b7f9695 => generated 275 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:00.865591 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6876043c-0294-48e7-a29a-efd545c912b6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:00.927200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f39a704-b580-4d72-85e1-b8b92497fd45 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:00.929220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f39a704-b580-4d72-85e1-b8b92497fd45 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:00.944844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:00.982295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6876043c-0294-48e7-a29a-efd545c912b6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:01.028708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea78cfb-4dbc-45c3-a28d-4d9d73738a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:01.029738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea78cfb-4dbc-45c3-a28d-4d9d73738a79 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:01.030229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea78cfb-4dbc-45c3-a28d-4d9d73738a79 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:01.031152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea78cfb-4dbc-45c3-a28d-4d9d73738a79 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:01.032571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea78cfb-4dbc-45c3-a28d-4d9d73738a79 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:01.034730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea78cfb-4dbc-45c3-a28d-4d9d73738a79 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:01.036873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 63/126] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:00 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:01.037979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea78cfb-4dbc-45c3-a28d-4d9d73738a79 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:01.053486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 64/127] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:01.054623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f39a704-b580-4d72-85e1-b8b92497fd45 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:01.106488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:01.152782 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea78cfb-4dbc-45c3-a28d-4d9d73738a79 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:01.178938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f39a704-b580-4d72-85e1-b8b92497fd45 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:01.187335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca91c9b6-09b3-4fbf-80af-3a300d4af101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:01.188123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91c9b6-09b3-4fbf-80af-3a300d4af101 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:01.188627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91c9b6-09b3-4fbf-80af-3a300d4af101 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:01.188935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91c9b6-09b3-4fbf-80af-3a300d4af101 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:01.199611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca91c9b6-09b3-4fbf-80af-3a300d4af101 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-ca91c9b6-09b3-4fbf-80af-3a300d4af101 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:01.205865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:01.206935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:01.207972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:01.209779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 64/128] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:01.215320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca91c9b6-09b3-4fbf-80af-3a300d4af101 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:01.221787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358930c8-2bcc-4e3c-a5e4-eb560e0b0ba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:01.222692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358930c8-2bcc-4e3c-a5e4-eb560e0b0ba4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:01.223114 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358930c8-2bcc-4e3c-a5e4-eb560e0b0ba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:01.223555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358930c8-2bcc-4e3c-a5e4-eb560e0b0ba4 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:01.224851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358930c8-2bcc-4e3c-a5e4-eb560e0b0ba4 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:01.226933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358930c8-2bcc-4e3c-a5e4-eb560e0b0ba4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:01.229519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 65/129] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:01 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:01.230603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358930c8-2bcc-4e3c-a5e4-eb560e0b0ba4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:01.235646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:01.335046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca91c9b6-09b3-4fbf-80af-3a300d4af101 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:01.368931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9401c5c7-5dcb-43b5-8300-b56f571835de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:01.369616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401c5c7-5dcb-43b5-8300-b56f571835de None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:01.369752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358930c8-2bcc-4e3c-a5e4-eb560e0b0ba4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:01.369877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401c5c7-5dcb-43b5-8300-b56f571835de None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:01.370180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401c5c7-5dcb-43b5-8300-b56f571835de None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/users/a734913c72ea45f3abe03c5a9b7f9695/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:01.372086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401c5c7-5dcb-43b5-8300-b56f571835de None admin] RBAC: Authorizing `identity:create_grant(project_id=1cf34cdf97b64f59bf465ae56dd45517, user_id=a734913c72ea45f3abe03c5a9b7f9695, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:01.373573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401c5c7-5dcb-43b5-8300-b56f571835de None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:01.384351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:01.393022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 65/130] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:16:01 2022] PUT /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/users/a734913c72ea45f3abe03c5a9b7f9695/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:01.394639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9401c5c7-5dcb-43b5-8300-b56f571835de None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:01.412652 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320a5761-0a49-4440-b87c-455410d2c971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:01.413361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320a5761-0a49-4440-b87c-455410d2c971 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:01.413764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320a5761-0a49-4440-b87c-455410d2c971 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:01.414040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320a5761-0a49-4440-b87c-455410d2c971 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:01.415646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320a5761-0a49-4440-b87c-455410d2c971 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:01.417477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320a5761-0a49-4440-b87c-455410d2c971 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:01.419571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 66/131] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:01 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:01.420414 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320a5761-0a49-4440-b87c-455410d2c971 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:01.481206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:01.538133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9401c5c7-5dcb-43b5-8300-b56f571835de None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:01.578625 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe6afc7-ec84-477d-9756-7b8c8a440ab1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:01.579260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe6afc7-ec84-477d-9756-7b8c8a440ab1 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:01.579539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe6afc7-ec84-477d-9756-7b8c8a440ab1 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:01.579813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe6afc7-ec84-477d-9756-7b8c8a440ab1 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:01.580662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe6afc7-ec84-477d-9756-7b8c8a440ab1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:01.581239 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe6afc7-ec84-477d-9756-7b8c8a440ab1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:01.582944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe6afc7-ec84-477d-9756-7b8c8a440ab1 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:01.593676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 66/132] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:01.594626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe6afc7-ec84-477d-9756-7b8c8a440ab1 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:01.672353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320a5761-0a49-4440-b87c-455410d2c971 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:01.712685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe6afc7-ec84-477d-9756-7b8c8a440ab1 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:01.732915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3900630-a6d6-462b-b94c-aedd0c8451a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:01.734337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3900630-a6d6-462b-b94c-aedd0c8451a0 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:01.734828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3900630-a6d6-462b-b94c-aedd0c8451a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:01.735301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3900630-a6d6-462b-b94c-aedd0c8451a0 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/users/a734913c72ea45f3abe03c5a9b7f9695/roles/74db8755b1f04d839260ccd3e36deb66` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:01.735402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:01.739648 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3900630-a6d6-462b-b94c-aedd0c8451a0 None admin] RBAC: Authorizing `identity:create_grant(project_id=1cf34cdf97b64f59bf465ae56dd45517, user_id=a734913c72ea45f3abe03c5a9b7f9695, role_id=74db8755b1f04d839260ccd3e36deb66)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:01.741368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3900630-a6d6-462b-b94c-aedd0c8451a0 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:01.758520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 67/133] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:16:01 2022] PUT /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/users/a734913c72ea45f3abe03c5a9b7f9695/roles/74db8755b1f04d839260ccd3e36deb66 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:01.759326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0ab4a9-67e8-49b1-a61f-ef62a29682f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:01.759971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0ab4a9-67e8-49b1-a61f-ef62a29682f4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:01.760291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0ab4a9-67e8-49b1-a61f-ef62a29682f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:01.760497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3900630-a6d6-462b-b94c-aedd0c8451a0 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:01.760616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0ab4a9-67e8-49b1-a61f-ef62a29682f4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:01.761501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0ab4a9-67e8-49b1-a61f-ef62a29682f4 None admin] RBAC: Adding query filter params (role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517, user.id=85841b4c5e464905b7f6869d24a1e483) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:01.761817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0ab4a9-67e8-49b1-a61f-ef62a29682f4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:01.763392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0ab4a9-67e8-49b1-a61f-ef62a29682f4 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:01.770282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 67/134] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:01 2022] GET /identity/v3/role_assignments?user.id=85841b4c5e464905b7f6869d24a1e483&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:01.811354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:01.883799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3900630-a6d6-462b-b94c-aedd0c8451a0 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:01.922718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b676970-fc27-4ea1-908a-641fa8ba643c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:01.923748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b676970-fc27-4ea1-908a-641fa8ba643c None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:01.924223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b676970-fc27-4ea1-908a-641fa8ba643c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:01.924732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b676970-fc27-4ea1-908a-641fa8ba643c None admin] PATH_INFO: `/v3/users/a734913c72ea45f3abe03c5a9b7f9695` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:01.926176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b676970-fc27-4ea1-908a-641fa8ba643c None admin] RBAC: Authorizing `identity:get_user(user_id=a734913c72ea45f3abe03c5a9b7f9695)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:01.927851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b676970-fc27-4ea1-908a-641fa8ba643c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:01.929740 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 68/135] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:01 2022] GET /identity/v3/users/a734913c72ea45f3abe03c5a9b7f9695 => generated 275 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:01.935415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0ab4a9-67e8-49b1-a61f-ef62a29682f4 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:02.082351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0ab4a9-67e8-49b1-a61f-ef62a29682f4 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:02.127993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d806af0-4d8d-4023-964f-a0a573d31d6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:02.128665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d806af0-4d8d-4023-964f-a0a573d31d6f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:02.128944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d806af0-4d8d-4023-964f-a0a573d31d6f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:02.129214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d806af0-4d8d-4023-964f-a0a573d31d6f None admin] PATH_INFO: `/v3/projects/e38dc6d96ea24fb296893d8dfb010287` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:02.130380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d806af0-4d8d-4023-964f-a0a573d31d6f None admin] RBAC: Authorizing `identity:get_project(project_id=e38dc6d96ea24fb296893d8dfb010287)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:02.131838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d806af0-4d8d-4023-964f-a0a573d31d6f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:02.133691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 68/136] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:01 2022] GET /identity/v3/projects/e38dc6d96ea24fb296893d8dfb010287 => generated 312 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:02.139441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b676970-fc27-4ea1-908a-641fa8ba643c None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:02.267376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b676970-fc27-4ea1-908a-641fa8ba643c None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:02.275230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:02.305258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a245569-92e6-4f49-84f4-970f608e8160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:02.305939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a245569-92e6-4f49-84f4-970f608e8160 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:02.306248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a245569-92e6-4f49-84f4-970f608e8160 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:02.306528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a245569-92e6-4f49-84f4-970f608e8160 None admin] PATH_INFO: `/v3/projects/e38dc6d96ea24fb296893d8dfb010287/users/a734913c72ea45f3abe03c5a9b7f9695/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:02.308524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a245569-92e6-4f49-84f4-970f608e8160 None admin] RBAC: Authorizing `identity:create_grant(project_id=e38dc6d96ea24fb296893d8dfb010287, user_id=a734913c72ea45f3abe03c5a9b7f9695, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:02.309986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a245569-92e6-4f49-84f4-970f608e8160 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:02.327069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 69/137] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:16:02 2022] PUT /identity/v3/projects/e38dc6d96ea24fb296893d8dfb010287/users/a734913c72ea45f3abe03c5a9b7f9695/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:02.600773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa86b9d-1761-4598-86bc-15947e59c8a2 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:02.601300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa86b9d-1761-4598-86bc-15947e59c8a2 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:02.601407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa86b9d-1761-4598-86bc-15947e59c8a2 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:02.602744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 69/138] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:02.608132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e119b-a6be-4f61-8e52-77e29fda38e1 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:02.608686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e119b-a6be-4f61-8e52-77e29fda38e1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:02.608762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e119b-a6be-4f61-8e52-77e29fda38e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:02.632403 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-100e119b-a6be-4f61-8e52-77e29fda38e1 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:02.633665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-100e119b-a6be-4f61-8e52-77e29fda38e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:02.728422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 70/139] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:03.012286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa86b9d-1761-4598-86bc-15947e59c8a2 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:03.077767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:03.147735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa86b9d-1761-4598-86bc-15947e59c8a2 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:03.189788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693aa566-f359-493e-90df-ac0dd7ce85c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:03.190459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693aa566-f359-493e-90df-ac0dd7ce85c5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:03.190739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693aa566-f359-493e-90df-ac0dd7ce85c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:03.191008 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693aa566-f359-493e-90df-ac0dd7ce85c5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:03.192099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693aa566-f359-493e-90df-ac0dd7ce85c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:03.193556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693aa566-f359-493e-90df-ac0dd7ce85c5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:03.195343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 70/140] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:03.201124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100e119b-a6be-4f61-8e52-77e29fda38e1 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:03.353125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:03.359716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100e119b-a6be-4f61-8e52-77e29fda38e1 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:03.412239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83c874d-916d-4c7a-98f1-53a3a2ab60a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:03.413221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83c874d-916d-4c7a-98f1-53a3a2ab60a1 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:03.413697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83c874d-916d-4c7a-98f1-53a3a2ab60a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:03.414187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83c874d-916d-4c7a-98f1-53a3a2ab60a1 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:03.415077 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83c874d-916d-4c7a-98f1-53a3a2ab60a1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:03.416712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83c874d-916d-4c7a-98f1-53a3a2ab60a1 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:03.453200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 71/141] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:16:03 2022] POST /identity/v3/users => generated 283 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:16:03.751584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc468753-f5f0-42b4-be36-42aa3d9d8610 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:03.752140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc468753-f5f0-42b4-be36-42aa3d9d8610 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:03.752140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc468753-f5f0-42b4-be36-42aa3d9d8610 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:03.753701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 71/142] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:16:03.759347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c5a920-467a-40ca-bc9c-64b19b3a07c0 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:03.760334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c5a920-467a-40ca-bc9c-64b19b3a07c0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:03.760806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c5a920-467a-40ca-bc9c-64b19b3a07c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:03.787115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-11c5a920-467a-40ca-bc9c-64b19b3a07c0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:03.788665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-11c5a920-467a-40ca-bc9c-64b19b3a07c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:03.868941 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 72/143] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:04.252725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc468753-f5f0-42b4-be36-42aa3d9d8610 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:04.383722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc468753-f5f0-42b4-be36-42aa3d9d8610 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:04.401334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:04.420054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0ee767-b675-4527-b557-0e6e41eb5ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:04.420704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ee767-b675-4527-b557-0e6e41eb5ec9 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:04.420981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ee767-b675-4527-b557-0e6e41eb5ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:04.421287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0ee767-b675-4527-b557-0e6e41eb5ec9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:04.433845 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0ee767-b675-4527-b557-0e6e41eb5ec9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-0c0ee767-b675-4527-b557-0e6e41eb5ec9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:04.439046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:04.439886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:04.440718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 72/144] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:16:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:04.447405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c5a920-467a-40ca-bc9c-64b19b3a07c0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:04.564224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:04.587836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c5a920-467a-40ca-bc9c-64b19b3a07c0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:04.635063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53703798-3cdc-4526-9f68-e66fc5e76c14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:04.635728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53703798-3cdc-4526-9f68-e66fc5e76c14 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:04.636008 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53703798-3cdc-4526-9f68-e66fc5e76c14 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:04.636288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53703798-3cdc-4526-9f68-e66fc5e76c14 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:04.636991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53703798-3cdc-4526-9f68-e66fc5e76c14 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:04.637301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53703798-3cdc-4526-9f68-e66fc5e76c14 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:04.638889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53703798-3cdc-4526-9f68-e66fc5e76c14 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:04.655272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 73/145] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:04.663098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0ee767-b675-4527-b557-0e6e41eb5ec9 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:04.791157 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0ee767-b675-4527-b557-0e6e41eb5ec9 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:04.824190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007165a2-58b6-427b-9be6-dde72845ae1d None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:04.824669 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007165a2-58b6-427b-9be6-dde72845ae1d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:04.824669 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007165a2-58b6-427b-9be6-dde72845ae1d None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:04.826184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 74/146] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:04.832504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03432b19-256f-467a-a772-3dd44bbc69ac None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:04.832783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03432b19-256f-467a-a772-3dd44bbc69ac None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:04.833061 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03432b19-256f-467a-a772-3dd44bbc69ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:04.841081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ea0062-a962-4140-83d6-6fea8377ee54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:04.841794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ea0062-a962-4140-83d6-6fea8377ee54 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:04.842075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ea0062-a962-4140-83d6-6fea8377ee54 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:04.847422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ea0062-a962-4140-83d6-6fea8377ee54 None admin] PATH_INFO: `/v3/users/d29c3b9b96c24a7daf5c64be528badeb` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:04.849653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ea0062-a962-4140-83d6-6fea8377ee54 None admin] RBAC: Authorizing `identity:get_user(user_id=d29c3b9b96c24a7daf5c64be528badeb)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:04.851245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ea0062-a962-4140-83d6-6fea8377ee54 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:04.855234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 73/147] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:04 2022] GET /identity/v3/users/d29c3b9b96c24a7daf5c64be528badeb => generated 247 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:04.858390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ea0062-a962-4140-83d6-6fea8377ee54 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:04.858859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-03432b19-256f-467a-a772-3dd44bbc69ac None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:04.860481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-03432b19-256f-467a-a772-3dd44bbc69ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:04.943168 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:04.970865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 75/148] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:04.972238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10ee3fa-0eab-4d9a-9a41-b3124a8dc8dd None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:04.972581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10ee3fa-0eab-4d9a-9a41-b3124a8dc8dd None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:04.972854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10ee3fa-0eab-4d9a-9a41-b3124a8dc8dd None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:04.974175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 76/149] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:04.979532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a37b69b-80b7-4f58-ac8f-2f78295221fd None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:04.979814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a37b69b-80b7-4f58-ac8f-2f78295221fd None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:04.980085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a37b69b-80b7-4f58-ac8f-2f78295221fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:05.010458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0a37b69b-80b7-4f58-ac8f-2f78295221fd None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:05.011663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0a37b69b-80b7-4f58-ac8f-2f78295221fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:05.031316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ea0062-a962-4140-83d6-6fea8377ee54 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:05.092670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d5843-2372-4591-b8a3-89f05a103f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:05.093443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d5843-2372-4591-b8a3-89f05a103f78 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:05.093775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d5843-2372-4591-b8a3-89f05a103f78 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:05.094054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d5843-2372-4591-b8a3-89f05a103f78 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:05.154818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 77/150] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:05.210814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60d5843-2372-4591-b8a3-89f05a103f78 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:05.212286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60d5843-2372-4591-b8a3-89f05a103f78 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:05.282602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:05.342372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 74/151] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 484 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:05.348341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a37b69b-80b7-4f58-ac8f-2f78295221fd None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:05.358917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:05.375379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d5843-2372-4591-b8a3-89f05a103f78 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:05.544278 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d5843-2372-4591-b8a3-89f05a103f78 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:05.575757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a37b69b-80b7-4f58-ac8f-2f78295221fd None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:05.639530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5836952-d500-4ffe-8a40-e64e5e808583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:05.640190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5836952-d500-4ffe-8a40-e64e5e808583 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:05.640564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5836952-d500-4ffe-8a40-e64e5e808583 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:05.640848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5836952-d500-4ffe-8a40-e64e5e808583 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:05.650975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575709b5-1a87-4adf-8fa2-4d23df8c6a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:05.651651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575709b5-1a87-4adf-8fa2-4d23df8c6a08 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:05.651939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575709b5-1a87-4adf-8fa2-4d23df8c6a08 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:05.652170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5836952-d500-4ffe-8a40-e64e5e808583 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:05.652271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575709b5-1a87-4adf-8fa2-4d23df8c6a08 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-c5836952-d500-4ffe-8a40-e64e5e808583 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:05.657068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:05.657919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:05.659135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:05.659711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575709b5-1a87-4adf-8fa2-4d23df8c6a08 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:05.659781 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 75/152] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:05 2022] GET /identity/v3/roles/reader => generated 84 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:16:05.659843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575709b5-1a87-4adf-8fa2-4d23df8c6a08 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:05.659905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbb35aa-cd03-4096-8bfb-04deda67520d None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:05.659967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbb35aa-cd03-4096-8bfb-04deda67520d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:05.660231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbb35aa-cd03-4096-8bfb-04deda67520d None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:05.660530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 78/153] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:05 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:05.661306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575709b5-1a87-4adf-8fa2-4d23df8c6a08 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:05.662022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 76/154] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:05.668117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbb35aa-cd03-4096-8bfb-04deda67520d None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:05.673510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:05.775949 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbb35aa-cd03-4096-8bfb-04deda67520d None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:05.798198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:05.812574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f61c718-82aa-414c-85bf-c5ff0d919d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:05.813335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f61c718-82aa-414c-85bf-c5ff0d919d12 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:05.813708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f61c718-82aa-414c-85bf-c5ff0d919d12 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:05.814076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f61c718-82aa-414c-85bf-c5ff0d919d12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:05.814905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f61c718-82aa-414c-85bf-c5ff0d919d12 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517, user.id=d29c3b9b96c24a7daf5c64be528badeb) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:05.815297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f61c718-82aa-414c-85bf-c5ff0d919d12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:05.816844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f61c718-82aa-414c-85bf-c5ff0d919d12 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:05.823157 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 77/155] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:05 2022] GET /identity/v3/role_assignments?user.id=d29c3b9b96c24a7daf5c64be528badeb&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=22819414415942d399db534708fe7103 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:05.824074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f61c718-82aa-414c-85bf-c5ff0d919d12 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:05.847981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575709b5-1a87-4adf-8fa2-4d23df8c6a08 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:05.913513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07a2180-a028-431f-8bd4-069b4b2ea048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:05.914202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07a2180-a028-431f-8bd4-069b4b2ea048 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:05.914541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07a2180-a028-431f-8bd4-069b4b2ea048 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:05.914815 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07a2180-a028-431f-8bd4-069b4b2ea048 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:05.936190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07a2180-a028-431f-8bd4-069b4b2ea048 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:05.938456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-b07a2180-a028-431f-8bd4-069b4b2ea048 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:05.951075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:05.952229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:05.953460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 79/156] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:05.955095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c398aa2-0d32-474b-a5d2-1c2005ebfd2c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:05.955095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c398aa2-0d32-474b-a5d2-1c2005ebfd2c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:05.955095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c398aa2-0d32-474b-a5d2-1c2005ebfd2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:05.982420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-3c398aa2-0d32-474b-a5d2-1c2005ebfd2c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:05.983670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3c398aa2-0d32-474b-a5d2-1c2005ebfd2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:05.986005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f61c718-82aa-414c-85bf-c5ff0d919d12 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:06.022022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b972a8a-bf1f-40e4-8b54-140907a09439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:06.022694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b972a8a-bf1f-40e4-8b54-140907a09439 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:06.022979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b972a8a-bf1f-40e4-8b54-140907a09439 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:06.023252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b972a8a-bf1f-40e4-8b54-140907a09439 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:06.023934 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b972a8a-bf1f-40e4-8b54-140907a09439 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:06.024236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b972a8a-bf1f-40e4-8b54-140907a09439 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:06.025667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b972a8a-bf1f-40e4-8b54-140907a09439 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:06.035121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 78/157] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:06.036001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b972a8a-bf1f-40e4-8b54-140907a09439 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:06.119209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 80/158] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:06.120240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c398aa2-0d32-474b-a5d2-1c2005ebfd2c None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:06.249242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b972a8a-bf1f-40e4-8b54-140907a09439 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:06.250031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:06.310032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:06.316544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c398aa2-0d32-474b-a5d2-1c2005ebfd2c None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:06.317472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0c52be-ba5c-4586-a6a5-0eef6771d9b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:06.318125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0c52be-ba5c-4586-a6a5-0eef6771d9b6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:06.318407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0c52be-ba5c-4586-a6a5-0eef6771d9b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:06.318686 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0c52be-ba5c-4586-a6a5-0eef6771d9b6 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:06.319366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0c52be-ba5c-4586-a6a5-0eef6771d9b6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:06.319671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0c52be-ba5c-4586-a6a5-0eef6771d9b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:06.321083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0c52be-ba5c-4586-a6a5-0eef6771d9b6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:06.333671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 79/159] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:06.334985 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80af1bf0-b1d6-4e4a-be6f-2d9d2cf78e57 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:06.335266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80af1bf0-b1d6-4e4a-be6f-2d9d2cf78e57 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:06.335537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80af1bf0-b1d6-4e4a-be6f-2d9d2cf78e57 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:06.336734 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 80/160] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:06.343068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80af1bf0-b1d6-4e4a-be6f-2d9d2cf78e57 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:06.383447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ac2fd1-4b4c-4412-997c-8b4fdd55eb86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:06.384124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac2fd1-4b4c-4412-997c-8b4fdd55eb86 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:06.384452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac2fd1-4b4c-4412-997c-8b4fdd55eb86 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:06.384736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac2fd1-4b4c-4412-997c-8b4fdd55eb86 None admin] PATH_INFO: `/v3/users/85841b4c5e464905b7f6869d24a1e483` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:06.390846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ac2fd1-4b4c-4412-997c-8b4fdd55eb86 None admin] RBAC: Authorizing `identity:get_user(user_id=85841b4c5e464905b7f6869d24a1e483)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:06.392354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ac2fd1-4b4c-4412-997c-8b4fdd55eb86 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:06.394023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 81/161] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:06 2022] GET /identity/v3/users/85841b4c5e464905b7f6869d24a1e483 => generated 289 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:06.395285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511846bd-5e5d-47fc-8273-bc0c824ea4ee None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:06.395567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511846bd-5e5d-47fc-8273-bc0c824ea4ee None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:06.395838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511846bd-5e5d-47fc-8273-bc0c824ea4ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:06.426219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-511846bd-5e5d-47fc-8273-bc0c824ea4ee None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:06.427431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-511846bd-5e5d-47fc-8273-bc0c824ea4ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:06.491713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80af1bf0-b1d6-4e4a-be6f-2d9d2cf78e57 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:06.544417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31905bb-096c-43af-9355-caed081795eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:06.544981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31905bb-096c-43af-9355-caed081795eb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:06.545262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31905bb-096c-43af-9355-caed081795eb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:06.545558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31905bb-096c-43af-9355-caed081795eb None admin] PATH_INFO: `/v3/users/a734913c72ea45f3abe03c5a9b7f9695` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:06.547996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31905bb-096c-43af-9355-caed081795eb None admin] RBAC: Authorizing `identity:get_user(user_id=a734913c72ea45f3abe03c5a9b7f9695)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:06.549474 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31905bb-096c-43af-9355-caed081795eb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:06.551293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 81/162] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:06 2022] GET /identity/v3/users/a734913c72ea45f3abe03c5a9b7f9695 => generated 275 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:06.555743 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31905bb-096c-43af-9355-caed081795eb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:06.583172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 82/163] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:06.584048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511846bd-5e5d-47fc-8273-bc0c824ea4ee None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:06.712405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:06.718582 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:06.797453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511846bd-5e5d-47fc-8273-bc0c824ea4ee None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:06.821822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31905bb-096c-43af-9355-caed081795eb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:06.865758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e810e070-93f9-49cd-994e-a47aece14d97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:06.867099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810e070-93f9-49cd-994e-a47aece14d97 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:06.867563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810e070-93f9-49cd-994e-a47aece14d97 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:06.868026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810e070-93f9-49cd-994e-a47aece14d97 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:06.902810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ffb97f-6076-4d25-a519-47bc1febb3dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:06.903469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ffb97f-6076-4d25-a519-47bc1febb3dd None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:06.903749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ffb97f-6076-4d25-a519-47bc1febb3dd None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:06.904021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ffb97f-6076-4d25-a519-47bc1febb3dd None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:06.905195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ffb97f-6076-4d25-a519-47bc1febb3dd None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:06.906869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ffb97f-6076-4d25-a519-47bc1febb3dd None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:06.913098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 82/164] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:06 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 357 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:06.913098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ffb97f-6076-4d25-a519-47bc1febb3dd None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:07.000159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e810e070-93f9-49cd-994e-a47aece14d97 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:07.001981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e810e070-93f9-49cd-994e-a47aece14d97 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:07.124447 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ffb97f-6076-4d25-a519-47bc1febb3dd None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:07.138894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:07.185544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:07.203403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4688d69-a638-46b8-9b6a-1ac5119d0cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:07.204578 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4688d69-a638-46b8-9b6a-1ac5119d0cf6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:07.205051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4688d69-a638-46b8-9b6a-1ac5119d0cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:07.205701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4688d69-a638-46b8-9b6a-1ac5119d0cf6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:07.222023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 83/165] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 638 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:07.223939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e810e070-93f9-49cd-994e-a47aece14d97 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:07.231865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4688d69-a638-46b8-9b6a-1ac5119d0cf6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-c4688d69-a638-46b8-9b6a-1ac5119d0cf6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:07.240430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:07.241293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:07.242359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:07.250612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 83/166] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 342 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:07.253085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4688d69-a638-46b8-9b6a-1ac5119d0cf6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:07.391998 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4688d69-a638-46b8-9b6a-1ac5119d0cf6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:07.429020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ba0ea2-9bf8-4c6a-b337-61a6f4eb7072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:07.429664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ba0ea2-9bf8-4c6a-b337-61a6f4eb7072 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:07.429942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ba0ea2-9bf8-4c6a-b337-61a6f4eb7072 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:07.430241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ba0ea2-9bf8-4c6a-b337-61a6f4eb7072 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:07.441820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ba0ea2-9bf8-4c6a-b337-61a6f4eb7072 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-e0ba0ea2-9bf8-4c6a-b337-61a6f4eb7072 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:07.447261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:07.448558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 84/167] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:07.449764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ba0ea2-9bf8-4c6a-b337-61a6f4eb7072 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:07.496563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e810e070-93f9-49cd-994e-a47aece14d97 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:07.509144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:07.561486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e193662b-f189-4699-a386-df8ecf7e9179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:07.562402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e193662b-f189-4699-a386-df8ecf7e9179 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:07.562680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e193662b-f189-4699-a386-df8ecf7e9179 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:07.562898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ba0ea2-9bf8-4c6a-b337-61a6f4eb7072 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:07.563003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e193662b-f189-4699-a386-df8ecf7e9179 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/users/85841b4c5e464905b7f6869d24a1e483/roles/0ff8d8f3f5ae426197cdffd9ea41a4e3` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:07.564899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e193662b-f189-4699-a386-df8ecf7e9179 None admin] RBAC: Authorizing `identity:create_grant(project_id=1cf34cdf97b64f59bf465ae56dd45517, user_id=85841b4c5e464905b7f6869d24a1e483, role_id=0ff8d8f3f5ae426197cdffd9ea41a4e3)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:07.566388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e193662b-f189-4699-a386-df8ecf7e9179 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:07.597446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858c2cf7-4877-4d1d-a367-45048e1342e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:07.598172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858c2cf7-4877-4d1d-a367-45048e1342e8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:07.598453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858c2cf7-4877-4d1d-a367-45048e1342e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:07.598723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858c2cf7-4877-4d1d-a367-45048e1342e8 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:07.599875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858c2cf7-4877-4d1d-a367-45048e1342e8 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:07.601515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858c2cf7-4877-4d1d-a367-45048e1342e8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:07.603376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 85/168] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:07 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:07.604146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858c2cf7-4877-4d1d-a367-45048e1342e8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:07.610037 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 84/169] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:16:07 2022] PUT /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/users/85841b4c5e464905b7f6869d24a1e483/roles/0ff8d8f3f5ae426197cdffd9ea41a4e3 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:07.610924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e193662b-f189-4699-a386-df8ecf7e9179 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:07.690082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:07.715542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858c2cf7-4877-4d1d-a367-45048e1342e8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:07.734910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:07.772419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd8942-af2f-4bea-9152-56b762c576b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:07.773072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd8942-af2f-4bea-9152-56b762c576b8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:07.773351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd8942-af2f-4bea-9152-56b762c576b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:07.773620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fd8942-af2f-4bea-9152-56b762c576b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:07.774358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fd8942-af2f-4bea-9152-56b762c576b8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:07.774661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fd8942-af2f-4bea-9152-56b762c576b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:07.776124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fd8942-af2f-4bea-9152-56b762c576b8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:07.794501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 86/170] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:07.795395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd8942-af2f-4bea-9152-56b762c576b8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:07.886390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e193662b-f189-4699-a386-df8ecf7e9179 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:07.964080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166d2479-ed7e-4057-93e6-2c57f17e5135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:07.964638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166d2479-ed7e-4057-93e6-2c57f17e5135 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:07.964917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166d2479-ed7e-4057-93e6-2c57f17e5135 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:07.965185 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166d2479-ed7e-4057-93e6-2c57f17e5135 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:07.965873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166d2479-ed7e-4057-93e6-2c57f17e5135 None admin] RBAC: Adding query filter params (name=member) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:07.966220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166d2479-ed7e-4057-93e6-2c57f17e5135 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:07.967612 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166d2479-ed7e-4057-93e6-2c57f17e5135 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:07.974838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fd8942-af2f-4bea-9152-56b762c576b8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:07.979344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 85/171] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 369 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:07.980206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166d2479-ed7e-4057-93e6-2c57f17e5135 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:08.026173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a431594a-2bad-43db-b605-ba27696cb66a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:08.026799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a431594a-2bad-43db-b605-ba27696cb66a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.027076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a431594a-2bad-43db-b605-ba27696cb66a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.027344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a431594a-2bad-43db-b605-ba27696cb66a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.028186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a431594a-2bad-43db-b605-ba27696cb66a None admin] RBAC: Adding query filter params (role.id=7a7e816668a7493bad0752baca37934c, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517, user.id=a734913c72ea45f3abe03c5a9b7f9695) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:08.028495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a431594a-2bad-43db-b605-ba27696cb66a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:08.030091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a431594a-2bad-43db-b605-ba27696cb66a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:08.038472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 87/172] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:07 2022] GET /identity/v3/role_assignments?user.id=a734913c72ea45f3abe03c5a9b7f9695&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=7a7e816668a7493bad0752baca37934c => generated 608 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:08.040174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22588d68-7fe0-4b7f-9123-438d8b9fe853 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.040459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22588d68-7fe0-4b7f-9123-438d8b9fe853 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.040716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22588d68-7fe0-4b7f-9123-438d8b9fe853 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.042113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 88/173] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:08.042817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22588d68-7fe0-4b7f-9123-438d8b9fe853 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:08.096850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166d2479-ed7e-4057-93e6-2c57f17e5135 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:08.107196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:08.131523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:08.146851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a0cbf4-ab46-40a7-8df3-599f2ba9b169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:08.147514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a0cbf4-ab46-40a7-8df3-599f2ba9b169 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.147791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a0cbf4-ab46-40a7-8df3-599f2ba9b169 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.148063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a0cbf4-ab46-40a7-8df3-599f2ba9b169 None admin] PATH_INFO: `/v3/users/a734913c72ea45f3abe03c5a9b7f9695` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.150222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a0cbf4-ab46-40a7-8df3-599f2ba9b169 None admin] RBAC: Authorizing `identity:get_user(user_id=a734913c72ea45f3abe03c5a9b7f9695)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:08.151990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a0cbf4-ab46-40a7-8df3-599f2ba9b169 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:08.153631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 86/174] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:07 2022] GET /identity/v3/users/a734913c72ea45f3abe03c5a9b7f9695 => generated 275 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:08.159526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7b9774-ba6f-47e7-957a-1cc909d6b40b None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.159526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7b9774-ba6f-47e7-957a-1cc909d6b40b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.159799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7b9774-ba6f-47e7-957a-1cc909d6b40b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.186143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0a7b9774-ba6f-47e7-957a-1cc909d6b40b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:08.186143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0a7b9774-ba6f-47e7-957a-1cc909d6b40b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:08.244952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22588d68-7fe0-4b7f-9123-438d8b9fe853 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:08.297821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 87/175] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:08.297821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7b9774-ba6f-47e7-957a-1cc909d6b40b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:08.328873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946a0019-f9c5-4dd7-814d-8ac7e1b85ae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:08.329920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946a0019-f9c5-4dd7-814d-8ac7e1b85ae5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.330409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946a0019-f9c5-4dd7-814d-8ac7e1b85ae5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.330947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946a0019-f9c5-4dd7-814d-8ac7e1b85ae5 None admin] PATH_INFO: `/v3/users/a734913c72ea45f3abe03c5a9b7f9695` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.334954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946a0019-f9c5-4dd7-814d-8ac7e1b85ae5 None admin] RBAC: Authorizing `identity:get_user(user_id=a734913c72ea45f3abe03c5a9b7f9695)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:08.336754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946a0019-f9c5-4dd7-814d-8ac7e1b85ae5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:08.339109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 89/176] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:08 2022] GET /identity/v3/users/a734913c72ea45f3abe03c5a9b7f9695 => generated 275 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:08.344912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946a0019-f9c5-4dd7-814d-8ac7e1b85ae5 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:08.455604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:08.467037 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7b9774-ba6f-47e7-957a-1cc909d6b40b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:08.471024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946a0019-f9c5-4dd7-814d-8ac7e1b85ae5 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:08.484232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:08.514314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcc5bd3-77d8-4933-b92c-cf717bf74f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:08.514913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcc5bd3-77d8-4933-b92c-cf717bf74f81 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.515191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcc5bd3-77d8-4933-b92c-cf717bf74f81 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.515458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcc5bd3-77d8-4933-b92c-cf717bf74f81 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.520992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805539f6-f113-4292-a01f-08116e24ea55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:08.521620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805539f6-f113-4292-a01f-08116e24ea55 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.521948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805539f6-f113-4292-a01f-08116e24ea55 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.522274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805539f6-f113-4292-a01f-08116e24ea55 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.597231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcc5bd3-77d8-4933-b92c-cf717bf74f81 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:08.599150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcc5bd3-77d8-4933-b92c-cf717bf74f81 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:08.620771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805539f6-f113-4292-a01f-08116e24ea55 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:08.622603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805539f6-f113-4292-a01f-08116e24ea55 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:08.706780 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:08.730219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 88/177] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:08.731654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d603d5-442f-41e3-9b08-7fd55b382af9 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.731824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d603d5-442f-41e3-9b08-7fd55b382af9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.732093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d603d5-442f-41e3-9b08-7fd55b382af9 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.737581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 89/178] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:08.737581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326ad6d-6ac3-4c8a-8a63-0458dab0d337 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.737581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326ad6d-6ac3-4c8a-8a63-0458dab0d337 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.737581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326ad6d-6ac3-4c8a-8a63-0458dab0d337 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.737581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 90/179] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:08.737581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326ad6d-6ac3-4c8a-8a63-0458dab0d337 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:08.754598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 90/180] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 15:16:08.755574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805539f6-f113-4292-a01f-08116e24ea55 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:08.768547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:08.849711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326ad6d-6ac3-4c8a-8a63-0458dab0d337 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:08.856233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805539f6-f113-4292-a01f-08116e24ea55 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:08.887519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c3b775-6b05-4a8d-9d16-3d50fa27dd15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:08.888413 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c3b775-6b05-4a8d-9d16-3d50fa27dd15 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.888692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c3b775-6b05-4a8d-9d16-3d50fa27dd15 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.889049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c3b775-6b05-4a8d-9d16-3d50fa27dd15 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.892383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5f9de3-1f73-47ed-805d-3d04c528725b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:08.893025 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5f9de3-1f73-47ed-805d-3d04c528725b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.893301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5f9de3-1f73-47ed-805d-3d04c528725b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.893567 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5f9de3-1f73-47ed-805d-3d04c528725b None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.894885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5f9de3-1f73-47ed-805d-3d04c528725b None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:08.896379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5f9de3-1f73-47ed-805d-3d04c528725b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:08.898158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 91/181] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:08 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:08.899191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c53a4-0d57-4c26-b5c3-b5b05ff4d233 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.899471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c53a4-0d57-4c26-b5c3-b5b05ff4d233 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.899746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c53a4-0d57-4c26-b5c3-b5b05ff4d233 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.900135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c3b775-6b05-4a8d-9d16-3d50fa27dd15 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-39c3b775-6b05-4a8d-9d16-3d50fa27dd15 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:08.905407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:08.906389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:08.907495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 91/182] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:16:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:08.908902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f289e887-d4d9-4832-9f47-e6e2ae24f002 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:08.909189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f289e887-d4d9-4832-9f47-e6e2ae24f002 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:08.909548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f289e887-d4d9-4832-9f47-e6e2ae24f002 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:08.924520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-fc0c53a4-0d57-4c26-b5c3-b5b05ff4d233 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:08.925720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fc0c53a4-0d57-4c26-b5c3-b5b05ff4d233 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:08.929161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-f289e887-d4d9-4832-9f47-e6e2ae24f002 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:08.930168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f289e887-d4d9-4832-9f47-e6e2ae24f002 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:08.969283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:08.995516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 92/183] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:08.996932 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0c53a4-0d57-4c26-b5c3-b5b05ff4d233 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:09.007261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:09.019545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 92/184] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:09.020821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f289e887-d4d9-4832-9f47-e6e2ae24f002 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:09.139264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f289e887-d4d9-4832-9f47-e6e2ae24f002 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:09.179984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bd1691-a266-45c4-86f8-93946e388636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:09.181245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bd1691-a266-45c4-86f8-93946e388636 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:09.181769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bd1691-a266-45c4-86f8-93946e388636 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:09.182269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bd1691-a266-45c4-86f8-93946e388636 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:09.183196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bd1691-a266-45c4-86f8-93946e388636 None admin] RBAC: Adding query filter params (role.id=74db8755b1f04d839260ccd3e36deb66, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517, user.id=a734913c72ea45f3abe03c5a9b7f9695) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:09.183697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bd1691-a266-45c4-86f8-93946e388636 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:09.185349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bd1691-a266-45c4-86f8-93946e388636 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:09.192667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 93/185] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:09 2022] GET /identity/v3/role_assignments?user.id=a734913c72ea45f3abe03c5a9b7f9695&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=74db8755b1f04d839260ccd3e36deb66 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:09.193836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bd1691-a266-45c4-86f8-93946e388636 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:09.239247 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:09.244474 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0c53a4-0d57-4c26-b5c3-b5b05ff4d233 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:09.294559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2835ab65-b781-49b7-acad-44977b3f3f43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:09.294916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2835ab65-b781-49b7-acad-44977b3f3f43 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:09.295156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2835ab65-b781-49b7-acad-44977b3f3f43 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:09.295422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2835ab65-b781-49b7-acad-44977b3f3f43 None admin] PATH_INFO: `/v3/projects/e38dc6d96ea24fb296893d8dfb010287` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:09.296652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2835ab65-b781-49b7-acad-44977b3f3f43 None admin] RBAC: Authorizing `identity:get_project(project_id=e38dc6d96ea24fb296893d8dfb010287)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:09.298475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2835ab65-b781-49b7-acad-44977b3f3f43 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:09.300628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 93/186] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:08 2022] GET /identity/v3/projects/e38dc6d96ea24fb296893d8dfb010287 => generated 312 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:09.301576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2835ab65-b781-49b7-acad-44977b3f3f43 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:09.329421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bd1691-a266-45c4-86f8-93946e388636 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:09.379720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab64a3-339d-407f-ba8d-f6edcfc25d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:09.381013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab64a3-339d-407f-ba8d-f6edcfc25d34 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:09.381524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab64a3-339d-407f-ba8d-f6edcfc25d34 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:09.382000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab64a3-339d-407f-ba8d-f6edcfc25d34 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:09.382910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cab64a3-339d-407f-ba8d-f6edcfc25d34 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:09.383423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cab64a3-339d-407f-ba8d-f6edcfc25d34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:09.385050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cab64a3-339d-407f-ba8d-f6edcfc25d34 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:09.396858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 94/187] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:09.398096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:09.398787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab64a3-339d-407f-ba8d-f6edcfc25d34 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:09.430738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2835ab65-b781-49b7-acad-44977b3f3f43 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:09.477163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00ceb79-6af4-4104-be83-30438214de92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:09.477915 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00ceb79-6af4-4104-be83-30438214de92 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:09.478219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00ceb79-6af4-4104-be83-30438214de92 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:09.478552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00ceb79-6af4-4104-be83-30438214de92 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:09.491175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00ceb79-6af4-4104-be83-30438214de92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-c00ceb79-6af4-4104-be83-30438214de92 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:09.497249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:09.498516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 94/188] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:09.499666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00ceb79-6af4-4104-be83-30438214de92 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:09.524783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:09.533222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab64a3-339d-407f-ba8d-f6edcfc25d34 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:09.575355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4009abd3-132f-4c91-a92b-cd4b490e7ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:09.575991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4009abd3-132f-4c91-a92b-cd4b490e7ff4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:09.576613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4009abd3-132f-4c91-a92b-cd4b490e7ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:09.576959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4009abd3-132f-4c91-a92b-cd4b490e7ff4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:09.578540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4009abd3-132f-4c91-a92b-cd4b490e7ff4 None admin] RBAC: Adding query filter params (role.id=7a7e816668a7493bad0752baca37934c, scope.project.id=e38dc6d96ea24fb296893d8dfb010287, user.id=a734913c72ea45f3abe03c5a9b7f9695) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:09.578857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4009abd3-132f-4c91-a92b-cd4b490e7ff4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:09.580337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4009abd3-132f-4c91-a92b-cd4b490e7ff4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:09.592446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 95/189] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:09 2022] GET /identity/v3/role_assignments?user.id=a734913c72ea45f3abe03c5a9b7f9695&scope.project.id=e38dc6d96ea24fb296893d8dfb010287&role.id=7a7e816668a7493bad0752baca37934c => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:09.594024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4009abd3-132f-4c91-a92b-cd4b490e7ff4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:09.637708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00ceb79-6af4-4104-be83-30438214de92 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:09.698490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dc3d17-68c1-4006-9c58-b35a8dfc4e30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:09.700948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dc3d17-68c1-4006-9c58-b35a8dfc4e30 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:09.701471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dc3d17-68c1-4006-9c58-b35a8dfc4e30 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:09.702546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dc3d17-68c1-4006-9c58-b35a8dfc4e30 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:09.705236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dc3d17-68c1-4006-9c58-b35a8dfc4e30 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:09.706775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dc3d17-68c1-4006-9c58-b35a8dfc4e30 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:09.708786 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 95/190] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:09.709646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dc3d17-68c1-4006-9c58-b35a8dfc4e30 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:09.717081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:09.749037 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4009abd3-132f-4c91-a92b-cd4b490e7ff4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:09.792400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb72a36-72b4-4831-a68b-d76575784726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:09.793206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb72a36-72b4-4831-a68b-d76575784726 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:09.793491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb72a36-72b4-4831-a68b-d76575784726 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:09.793759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb72a36-72b4-4831-a68b-d76575784726 None admin] PATH_INFO: `/v3/users/9c9dd70e89644f408a0d1bad859993f6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:09.810435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb72a36-72b4-4831-a68b-d76575784726 None admin] RBAC: Authorizing `identity:get_user(user_id=9c9dd70e89644f408a0d1bad859993f6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:09.811986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb72a36-72b4-4831-a68b-d76575784726 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:09.813991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 96/191] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:09 2022] GET /identity/v3/users/9c9dd70e89644f408a0d1bad859993f6 => generated 283 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:09.815316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb72a36-72b4-4831-a68b-d76575784726 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:09.837963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:09.846313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dc3d17-68c1-4006-9c58-b35a8dfc4e30 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:09.887382 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a5bf91-4d51-4c6b-8844-9dd37789434c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:09.888248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5bf91-4d51-4c6b-8844-9dd37789434c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:09.888666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5bf91-4d51-4c6b-8844-9dd37789434c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:09.889658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5bf91-4d51-4c6b-8844-9dd37789434c None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:09.890600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a5bf91-4d51-4c6b-8844-9dd37789434c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:09.891112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a5bf91-4d51-4c6b-8844-9dd37789434c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:09.892655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a5bf91-4d51-4c6b-8844-9dd37789434c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:09.905366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 96/192] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:09 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:09.906510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a5bf91-4d51-4c6b-8844-9dd37789434c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:09.945347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb72a36-72b4-4831-a68b-d76575784726 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:09.980897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62baeec-4f1c-447d-998f-ecb1e4677fe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:09.981539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62baeec-4f1c-447d-998f-ecb1e4677fe4 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:09.981809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62baeec-4f1c-447d-998f-ecb1e4677fe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:09.982081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62baeec-4f1c-447d-998f-ecb1e4677fe4 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:09.982787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62baeec-4f1c-447d-998f-ecb1e4677fe4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:09.984667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62baeec-4f1c-447d-998f-ecb1e4677fe4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:09.991794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:10.007758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 97/193] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:16:09 2022] POST /identity/v3/users => generated 297 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:16:10.008991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62baeec-4f1c-447d-998f-ecb1e4677fe4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:10.041803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a5bf91-4d51-4c6b-8844-9dd37789434c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:10.085349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30e8caa-b938-423a-9c68-fbfd3a657ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:10.086305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30e8caa-b938-423a-9c68-fbfd3a657ff4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:10.086730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30e8caa-b938-423a-9c68-fbfd3a657ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:10.087133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30e8caa-b938-423a-9c68-fbfd3a657ff4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:10.139249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:10.172768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62baeec-4f1c-447d-998f-ecb1e4677fe4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:10.177223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30e8caa-b938-423a-9c68-fbfd3a657ff4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:10.179023 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30e8caa-b938-423a-9c68-fbfd3a657ff4 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:10.212674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1db979-ca8c-4592-a2e4-bd5168bcd48f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:10.213323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1db979-ca8c-4592-a2e4-bd5168bcd48f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:10.213593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1db979-ca8c-4592-a2e4-bd5168bcd48f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:10.213858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1db979-ca8c-4592-a2e4-bd5168bcd48f None admin] PATH_INFO: `/v3/users/9c9dd70e89644f408a0d1bad859993f6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:10.215057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1db979-ca8c-4592-a2e4-bd5168bcd48f None admin] RBAC: Authorizing `identity:get_user(user_id=9c9dd70e89644f408a0d1bad859993f6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:10.217205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1db979-ca8c-4592-a2e4-bd5168bcd48f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:10.219234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 98/194] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:10 2022] GET /identity/v3/users/9c9dd70e89644f408a0d1bad859993f6 => generated 283 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:10.226870 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1db979-ca8c-4592-a2e4-bd5168bcd48f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:10.257477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:10.313352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 97/195] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:10.321530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30e8caa-b938-423a-9c68-fbfd3a657ff4 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:10.369089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1db979-ca8c-4592-a2e4-bd5168bcd48f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:10.416301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc778454-3cbf-4ee0-af47-3473fa110d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:10.416932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc778454-3cbf-4ee0-af47-3473fa110d96 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:10.417209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc778454-3cbf-4ee0-af47-3473fa110d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:10.417481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc778454-3cbf-4ee0-af47-3473fa110d96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:10.432406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:10.449591 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30e8caa-b938-423a-9c68-fbfd3a657ff4 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:10.491763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1d33f3-e398-4413-920a-6b4a4d1771ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:10.492283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1d33f3-e398-4413-920a-6b4a4d1771ee None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:10.492568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1d33f3-e398-4413-920a-6b4a4d1771ee None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:10.492840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1d33f3-e398-4413-920a-6b4a4d1771ee None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:10.494510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1d33f3-e398-4413-920a-6b4a4d1771ee None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:10.496136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1d33f3-e398-4413-920a-6b4a4d1771ee None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:10.497969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 98/196] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:10 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:10.502283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67875b08-b53a-43ff-a796-644fc0a1f177 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:10.502283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67875b08-b53a-43ff-a796-644fc0a1f177 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:10.502283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67875b08-b53a-43ff-a796-644fc0a1f177 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:10.502283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 99/197] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:10.504886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67875b08-b53a-43ff-a796-644fc0a1f177 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:10.505362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc778454-3cbf-4ee0-af47-3473fa110d96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:10.507029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc778454-3cbf-4ee0-af47-3473fa110d96 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:10.542906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:10.616719 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67875b08-b53a-43ff-a796-644fc0a1f177 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:10.645004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 99/198] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 13 15:16:10.649633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f4c6c9-d47e-499e-b572-85c303c2bfd2 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:10.649633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f4c6c9-d47e-499e-b572-85c303c2bfd2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:10.649633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f4c6c9-d47e-499e-b572-85c303c2bfd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:10.665178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8655b69e-5dc4-4a79-91ef-25a35301b321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:10.665802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8655b69e-5dc4-4a79-91ef-25a35301b321 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:10.666081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8655b69e-5dc4-4a79-91ef-25a35301b321 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:10.666453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8655b69e-5dc4-4a79-91ef-25a35301b321 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:10.667132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8655b69e-5dc4-4a79-91ef-25a35301b321 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=dea71e3f98bf423598a93def5ac1cd19, user.id=9c9dd70e89644f408a0d1bad859993f6) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:10.667440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8655b69e-5dc4-4a79-91ef-25a35301b321 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:10.668909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8655b69e-5dc4-4a79-91ef-25a35301b321 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:10.673790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-18f4c6c9-d47e-499e-b572-85c303c2bfd2 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:10.675073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 100/199] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:10 2022] GET /identity/v3/role_assignments?user.id=9c9dd70e89644f408a0d1bad859993f6&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=22819414415942d399db534708fe7103 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:10.675715 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8655b69e-5dc4-4a79-91ef-25a35301b321 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:10.676497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-18f4c6c9-d47e-499e-b572-85c303c2bfd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:10.743851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:10.808280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 100/200] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:10.846689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:10.850742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8655b69e-5dc4-4a79-91ef-25a35301b321 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:10.887343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2442712d-2406-4a20-9d7f-decb8e9b0b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:10.887983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2442712d-2406-4a20-9d7f-decb8e9b0b43 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:10.888264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2442712d-2406-4a20-9d7f-decb8e9b0b43 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:10.888534 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2442712d-2406-4a20-9d7f-decb8e9b0b43 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:10.889706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2442712d-2406-4a20-9d7f-decb8e9b0b43 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:10.891274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2442712d-2406-4a20-9d7f-decb8e9b0b43 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:10.893227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 101/201] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:10 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:10.899118 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f4c6c9-d47e-499e-b572-85c303c2bfd2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:11.008057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f4c6c9-d47e-499e-b572-85c303c2bfd2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:11.051749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906a98bf-2cb7-4dae-83df-4d10a9a283fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:11.052769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906a98bf-2cb7-4dae-83df-4d10a9a283fb None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:11.053239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906a98bf-2cb7-4dae-83df-4d10a9a283fb None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:11.053698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906a98bf-2cb7-4dae-83df-4d10a9a283fb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:11.054670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906a98bf-2cb7-4dae-83df-4d10a9a283fb None admin] RBAC: Adding query filter params (role.id=74db8755b1f04d839260ccd3e36deb66, scope.project.id=dea71e3f98bf423598a93def5ac1cd19, user.id=9c9dd70e89644f408a0d1bad859993f6) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:11.055173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906a98bf-2cb7-4dae-83df-4d10a9a283fb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:11.056809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906a98bf-2cb7-4dae-83df-4d10a9a283fb None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:11.065003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 101/202] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:10 2022] GET /identity/v3/role_assignments?user.id=9c9dd70e89644f408a0d1bad859993f6&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=74db8755b1f04d839260ccd3e36deb66 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:11.130688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2442712d-2406-4a20-9d7f-decb8e9b0b43 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:11.271054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2442712d-2406-4a20-9d7f-decb8e9b0b43 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:11.309614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc5eb26-ee9a-44fa-b4ef-98722f938106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:11.310292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc5eb26-ee9a-44fa-b4ef-98722f938106 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:11.310575 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc5eb26-ee9a-44fa-b4ef-98722f938106 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:11.310846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc5eb26-ee9a-44fa-b4ef-98722f938106 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:11.321892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc5eb26-ee9a-44fa-b4ef-98722f938106 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-3cc5eb26-ee9a-44fa-b4ef-98722f938106 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:11.327413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:11.328436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:11.329235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 102/203] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:11 2022] GET /identity/v3/roles/reader => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:11.334692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906a98bf-2cb7-4dae-83df-4d10a9a283fb None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:11.373124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:11.465477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906a98bf-2cb7-4dae-83df-4d10a9a283fb None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:11.499847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e37232-0bdc-4359-9cb8-3c716ecc5f0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:11.500467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e37232-0bdc-4359-9cb8-3c716ecc5f0d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:11.500747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e37232-0bdc-4359-9cb8-3c716ecc5f0d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:11.501022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e37232-0bdc-4359-9cb8-3c716ecc5f0d None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:11.501697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e37232-0bdc-4359-9cb8-3c716ecc5f0d None admin] RBAC: Adding query filter params (name=reader) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:11.502001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e37232-0bdc-4359-9cb8-3c716ecc5f0d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:11.503418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e37232-0bdc-4359-9cb8-3c716ecc5f0d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:11.511942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 102/204] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:11 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:11.517929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc5eb26-ee9a-44fa-b4ef-98722f938106 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:11.546142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:11.649392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc5eb26-ee9a-44fa-b4ef-98722f938106 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:11.701083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d9895c-cf63-4ee8-9de2-113bc7df3ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:11.701750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d9895c-cf63-4ee8-9de2-113bc7df3ed6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:11.702029 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d9895c-cf63-4ee8-9de2-113bc7df3ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:11.702444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d9895c-cf63-4ee8-9de2-113bc7df3ed6 None admin] PATH_INFO: `/v3/users/85841b4c5e464905b7f6869d24a1e483` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:11.703641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d9895c-cf63-4ee8-9de2-113bc7df3ed6 None admin] RBAC: Authorizing `identity:get_user(user_id=85841b4c5e464905b7f6869d24a1e483)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:11.705153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d9895c-cf63-4ee8-9de2-113bc7df3ed6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:11.706782 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 103/205] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:11 2022] GET /identity/v3/users/85841b4c5e464905b7f6869d24a1e483 => generated 289 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:11.712499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e37232-0bdc-4359-9cb8-3c716ecc5f0d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:11.834154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e37232-0bdc-4359-9cb8-3c716ecc5f0d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:11.843830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:11.869137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e5995e-0aa0-42c0-8083-acbf0c8ae2b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:11.869760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e5995e-0aa0-42c0-8083-acbf0c8ae2b4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:11.870042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e5995e-0aa0-42c0-8083-acbf0c8ae2b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:11.870347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e5995e-0aa0-42c0-8083-acbf0c8ae2b4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:11.938455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e5995e-0aa0-42c0-8083-acbf0c8ae2b4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:11.939960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e5995e-0aa0-42c0-8083-acbf0c8ae2b4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:12.036626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:12.041032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 103/206] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:12.051613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d9895c-cf63-4ee8-9de2-113bc7df3ed6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:12.200218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:12.207243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d9895c-cf63-4ee8-9de2-113bc7df3ed6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:12.242717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc99baa-8884-47ec-b59e-81ad3ffcdab1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:12.243362 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc99baa-8884-47ec-b59e-81ad3ffcdab1 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:12.243638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc99baa-8884-47ec-b59e-81ad3ffcdab1 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:12.243909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc99baa-8884-47ec-b59e-81ad3ffcdab1 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:12.245046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc99baa-8884-47ec-b59e-81ad3ffcdab1 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:12.246546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc99baa-8884-47ec-b59e-81ad3ffcdab1 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:12.248503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 104/207] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:12 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:12.255756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e5995e-0aa0-42c0-8083-acbf0c8ae2b4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:12.461241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e5995e-0aa0-42c0-8083-acbf0c8ae2b4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:12.511083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ec6515-6e14-4324-affa-401f901597b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:12.511708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ec6515-6e14-4324-affa-401f901597b5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:12.511981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ec6515-6e14-4324-affa-401f901597b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:12.512247 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ec6515-6e14-4324-affa-401f901597b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:12.512977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ec6515-6e14-4324-affa-401f901597b5 None admin] RBAC: Adding query filter params (role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517, user.id=85841b4c5e464905b7f6869d24a1e483) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:12.513281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ec6515-6e14-4324-affa-401f901597b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:12.514741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ec6515-6e14-4324-affa-401f901597b5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:12.521173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 104/208] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:12 2022] GET /identity/v3/role_assignments?user.id=85841b4c5e464905b7f6869d24a1e483&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3 => generated 608 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:13.027138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afc0ed7-f6b5-4f5b-acf4-9baa6563ab7c None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.027430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afc0ed7-f6b5-4f5b-acf4-9baa6563ab7c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.027813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afc0ed7-f6b5-4f5b-acf4-9baa6563ab7c None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.029228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 105/209] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:13.036860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1344359-fa21-450b-8d8d-1ed747d69d9c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.037331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1344359-fa21-450b-8d8d-1ed747d69d9c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.037425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1344359-fa21-450b-8d8d-1ed747d69d9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.060897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d1344359-fa21-450b-8d8d-1ed747d69d9c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:13.062613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d1344359-fa21-450b-8d8d-1ed747d69d9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:13.134639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:13.145808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 105/210] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:13.277504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed3629c-5c91-4e91-8be7-0059bd7e2196 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.278045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed3629c-5c91-4e91-8be7-0059bd7e2196 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.278164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed3629c-5c91-4e91-8be7-0059bd7e2196 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.279688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 106/211] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:13.285647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad38840-a2d4-40a1-8a8e-0bf5ec5bb46f None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.285987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad38840-a2d4-40a1-8a8e-0bf5ec5bb46f None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.286289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad38840-a2d4-40a1-8a8e-0bf5ec5bb46f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.305205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4ad38840-a2d4-40a1-8a8e-0bf5ec5bb46f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:13.306472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4ad38840-a2d4-40a1-8a8e-0bf5ec5bb46f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:13.379015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 106/212] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:13.426227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed3629c-5c91-4e91-8be7-0059bd7e2196 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:13.538565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed3629c-5c91-4e91-8be7-0059bd7e2196 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:13.572891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60278d3f-0972-46c7-8883-5077cc08edd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:13.573530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60278d3f-0972-46c7-8883-5077cc08edd7 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.573808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60278d3f-0972-46c7-8883-5077cc08edd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.574078 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60278d3f-0972-46c7-8883-5077cc08edd7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.584956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60278d3f-0972-46c7-8883-5077cc08edd7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-60278d3f-0972-46c7-8883-5077cc08edd7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:13.589933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:13.590914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:13.591688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 107/213] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:13.597660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad38840-a2d4-40a1-8a8e-0bf5ec5bb46f None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:13.667636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60278d3f-0972-46c7-8883-5077cc08edd7 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:13.704709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:13.712967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad38840-a2d4-40a1-8a8e-0bf5ec5bb46f None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:13.737738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:13.751244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69e26f8-d2b2-45f6-ac5a-ddf0ef12c14f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:13.751892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69e26f8-d2b2-45f6-ac5a-ddf0ef12c14f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.752151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69e26f8-d2b2-45f6-ac5a-ddf0ef12c14f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.752424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69e26f8-d2b2-45f6-ac5a-ddf0ef12c14f None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.753094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69e26f8-d2b2-45f6-ac5a-ddf0ef12c14f None admin] RBAC: Adding query filter params (name=member) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:13.753395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69e26f8-d2b2-45f6-ac5a-ddf0ef12c14f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:13.754816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69e26f8-d2b2-45f6-ac5a-ddf0ef12c14f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:13.763078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 107/214] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:13.764291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efa336-e7a8-4c66-a9fe-cd084a5e5243 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.764577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efa336-e7a8-4c66-a9fe-cd084a5e5243 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.764957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6efa336-e7a8-4c66-a9fe-cd084a5e5243 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.766249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 108/215] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:13.768192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e1cd1a-ddfc-40c3-8d34-7fec4fb882d5 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.768192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e1cd1a-ddfc-40c3-8d34-7fec4fb882d5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.768192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e1cd1a-ddfc-40c3-8d34-7fec4fb882d5 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.769230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 109/216] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:13 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:13.769925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e1cd1a-ddfc-40c3-8d34-7fec4fb882d5 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:13.781660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60278d3f-0972-46c7-8883-5077cc08edd7 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:13.815659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144e25c9-b553-46e9-9b1e-a8be596ca09f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:13.816230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144e25c9-b553-46e9-9b1e-a8be596ca09f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.816504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144e25c9-b553-46e9-9b1e-a8be596ca09f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.816762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144e25c9-b553-46e9-9b1e-a8be596ca09f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.817871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144e25c9-b553-46e9-9b1e-a8be596ca09f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:13.819374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144e25c9-b553-46e9-9b1e-a8be596ca09f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:13.820984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 108/217] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:13.826203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aeb9e4-33de-4db6-9c6b-76c9a6d2529b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.826465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aeb9e4-33de-4db6-9c6b-76c9a6d2529b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.826734 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aeb9e4-33de-4db6-9c6b-76c9a6d2529b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.845430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-12aeb9e4-33de-4db6-9c6b-76c9a6d2529b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:13.846678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-12aeb9e4-33de-4db6-9c6b-76c9a6d2529b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:13.878093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e1cd1a-ddfc-40c3-8d34-7fec4fb882d5 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:13.911443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245a8534-2de4-4c10-a28d-221baf730f2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:13.912070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a8534-2de4-4c10-a28d-221baf730f2f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.912346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a8534-2de4-4c10-a28d-221baf730f2f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.912617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a8534-2de4-4c10-a28d-221baf730f2f None admin] PATH_INFO: `/v3/users/8010bf3e93524f76998c82f0cc940491` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.914671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 109/218] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:13.915908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a7330c-ab06-442e-b62b-c48c580ff2e2 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:13.916187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a7330c-ab06-442e-b62b-c48c580ff2e2 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:13.916455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a7330c-ab06-442e-b62b-c48c580ff2e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:13.926120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245a8534-2de4-4c10-a28d-221baf730f2f None admin] RBAC: Authorizing `identity:get_user(user_id=8010bf3e93524f76998c82f0cc940491)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:13.927586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245a8534-2de4-4c10-a28d-221baf730f2f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:13.929232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 110/219] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:13 2022] GET /identity/v3/users/8010bf3e93524f76998c82f0cc940491 => generated 297 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:13.929967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245a8534-2de4-4c10-a28d-221baf730f2f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:13.951230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-91a7330c-ab06-442e-b62b-c48c580ff2e2 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:13.952804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-91a7330c-ab06-442e-b62b-c48c580ff2e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:13.968227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:13.974722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:14.027585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 110/220] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:14.035291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a7330c-ab06-442e-b62b-c48c580ff2e2 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:14.036628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245a8534-2de4-4c10-a28d-221baf730f2f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:14.075675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8bce55-8723-4027-9424-3c8674261a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:14.077129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8bce55-8723-4027-9424-3c8674261a94 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:14.077464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8bce55-8723-4027-9424-3c8674261a94 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:14.077696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8bce55-8723-4027-9424-3c8674261a94 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:14.078394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8bce55-8723-4027-9424-3c8674261a94 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:14.080114 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8bce55-8723-4027-9424-3c8674261a94 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:14.113200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 111/221] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:16:13 2022] POST /identity/v3/users => generated 297 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:16:14.164589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a7330c-ab06-442e-b62b-c48c580ff2e2 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:14.186516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8bce55-8723-4027-9424-3c8674261a94 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:14.207495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df85134-6012-4dc9-9044-4afa83657689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:14.208136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df85134-6012-4dc9-9044-4afa83657689 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:14.208588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df85134-6012-4dc9-9044-4afa83657689 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:14.208866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df85134-6012-4dc9-9044-4afa83657689 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:14.239507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:14.306990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:14.311257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8bce55-8723-4027-9424-3c8674261a94 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:14.311835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df85134-6012-4dc9-9044-4afa83657689 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:14.314004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df85134-6012-4dc9-9044-4afa83657689 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:14.353945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774bc5b4-d148-497e-9cb2-3a0e4f612cd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:14.354851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774bc5b4-d148-497e-9cb2-3a0e4f612cd9 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:14.355138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774bc5b4-d148-497e-9cb2-3a0e4f612cd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:14.355392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774bc5b4-d148-497e-9cb2-3a0e4f612cd9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:14.367468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774bc5b4-d148-497e-9cb2-3a0e4f612cd9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-774bc5b4-d148-497e-9cb2-3a0e4f612cd9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:14.372167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:14.373063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:14.374026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 112/222] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:16:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:16:14.379493 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774bc5b4-d148-497e-9cb2-3a0e4f612cd9 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:14.434528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 111/223] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:14.435541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df85134-6012-4dc9-9044-4afa83657689 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:14.493785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:14.502552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774bc5b4-d148-497e-9cb2-3a0e4f612cd9 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:14.542776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7193850-8a13-4538-90ef-33976effcecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:14.543674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7193850-8a13-4538-90ef-33976effcecd None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:14.544130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7193850-8a13-4538-90ef-33976effcecd None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:14.544564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7193850-8a13-4538-90ef-33976effcecd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:14.558457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df85134-6012-4dc9-9044-4afa83657689 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:14.559379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7193850-8a13-4538-90ef-33976effcecd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-f7193850-8a13-4538-90ef-33976effcecd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:14.565655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:14.566754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:14.567715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:14.569688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 113/224] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:14.570639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7193850-8a13-4538-90ef-33976effcecd None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:14.594193 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:14.604192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101bf62a-513b-41f1-a29f-040177857cb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:14.604874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101bf62a-513b-41f1-a29f-040177857cb0 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:14.605170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101bf62a-513b-41f1-a29f-040177857cb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:14.605429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101bf62a-513b-41f1-a29f-040177857cb0 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:14.606139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101bf62a-513b-41f1-a29f-040177857cb0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:14.606441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101bf62a-513b-41f1-a29f-040177857cb0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:14.608333 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101bf62a-513b-41f1-a29f-040177857cb0 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:14.620013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 112/225] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:14.620823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101bf62a-513b-41f1-a29f-040177857cb0 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:14.703689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7193850-8a13-4538-90ef-33976effcecd None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:14.749934 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101bf62a-513b-41f1-a29f-040177857cb0 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:14.753153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e84afd-4077-4e2f-b8eb-68c8886eded3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:14.754056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e84afd-4077-4e2f-b8eb-68c8886eded3 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:14.754567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e84afd-4077-4e2f-b8eb-68c8886eded3 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:14.755028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e84afd-4077-4e2f-b8eb-68c8886eded3 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:14.756435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e84afd-4077-4e2f-b8eb-68c8886eded3 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:14.758696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e84afd-4077-4e2f-b8eb-68c8886eded3 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:14.761614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 114/226] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:14 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:14.762947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e84afd-4077-4e2f-b8eb-68c8886eded3 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:14.771997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:14.796930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7933b40d-fd71-4bb7-9d25-103c75d0cdee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:14.797945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7933b40d-fd71-4bb7-9d25-103c75d0cdee None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:14.798554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7933b40d-fd71-4bb7-9d25-103c75d0cdee None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:14.799015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7933b40d-fd71-4bb7-9d25-103c75d0cdee None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:14.799946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7933b40d-fd71-4bb7-9d25-103c75d0cdee None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:14.800436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7933b40d-fd71-4bb7-9d25-103c75d0cdee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:14.802047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7933b40d-fd71-4bb7-9d25-103c75d0cdee None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:14.812424 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 113/227] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:14 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:14.813670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7933b40d-fd71-4bb7-9d25-103c75d0cdee None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:14.860816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:14.877905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e84afd-4077-4e2f-b8eb-68c8886eded3 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:14.920489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeab87a-447d-45f1-a49d-7abeefbfea3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:14.921167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeab87a-447d-45f1-a49d-7abeefbfea3e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:14.921443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeab87a-447d-45f1-a49d-7abeefbfea3e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:14.921952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeab87a-447d-45f1-a49d-7abeefbfea3e None admin] PATH_INFO: `/v3/users/9c9dd70e89644f408a0d1bad859993f6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:14.923657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeab87a-447d-45f1-a49d-7abeefbfea3e None admin] RBAC: Authorizing `identity:get_user(user_id=9c9dd70e89644f408a0d1bad859993f6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:14.925389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeab87a-447d-45f1-a49d-7abeefbfea3e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:14.927193 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 115/228] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:14 2022] GET /identity/v3/users/9c9dd70e89644f408a0d1bad859993f6 => generated 283 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:14.927716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7933b40d-fd71-4bb7-9d25-103c75d0cdee None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:14.930530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeab87a-447d-45f1-a49d-7abeefbfea3e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:14.968050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8997c849-36f6-4cca-ac53-538a013db54f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:14.969039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8997c849-36f6-4cca-ac53-538a013db54f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:14.969537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8997c849-36f6-4cca-ac53-538a013db54f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:14.970048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8997c849-36f6-4cca-ac53-538a013db54f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:14.971041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8997c849-36f6-4cca-ac53-538a013db54f None admin] RBAC: Adding query filter params (role.id=7a7e816668a7493bad0752baca37934c, scope.project.id=dea71e3f98bf423598a93def5ac1cd19, user.id=8010bf3e93524f76998c82f0cc940491) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:14.971539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8997c849-36f6-4cca-ac53-538a013db54f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:14.973201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8997c849-36f6-4cca-ac53-538a013db54f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:14.979947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 114/229] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:14 2022] GET /identity/v3/role_assignments?user.id=8010bf3e93524f76998c82f0cc940491&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=7a7e816668a7493bad0752baca37934c => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:14.981034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8997c849-36f6-4cca-ac53-538a013db54f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:15.080452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:15.093148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeab87a-447d-45f1-a49d-7abeefbfea3e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:15.131852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df562e5-5839-4f6f-b7db-f599691de2ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:15.132419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df562e5-5839-4f6f-b7db-f599691de2ba None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:15.132694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df562e5-5839-4f6f-b7db-f599691de2ba None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:15.132962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df562e5-5839-4f6f-b7db-f599691de2ba None admin] PATH_INFO: `/v3/users/9c9dd70e89644f408a0d1bad859993f6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:15.134526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df562e5-5839-4f6f-b7db-f599691de2ba None admin] RBAC: Authorizing `identity:get_user(user_id=9c9dd70e89644f408a0d1bad859993f6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:15.136253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df562e5-5839-4f6f-b7db-f599691de2ba None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:15.138308 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 116/230] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:14 2022] GET /identity/v3/users/9c9dd70e89644f408a0d1bad859993f6 => generated 283 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:15.150140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df562e5-5839-4f6f-b7db-f599691de2ba None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:15.174404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:15.243105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8997c849-36f6-4cca-ac53-538a013db54f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:15.265270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df562e5-5839-4f6f-b7db-f599691de2ba None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:15.283463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc9289d-61e3-4e03-982e-b0269c87f2b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:15.284660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc9289d-61e3-4e03-982e-b0269c87f2b5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:15.285137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc9289d-61e3-4e03-982e-b0269c87f2b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:15.285601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc9289d-61e3-4e03-982e-b0269c87f2b5 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:15.287156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc9289d-61e3-4e03-982e-b0269c87f2b5 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:15.288902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc9289d-61e3-4e03-982e-b0269c87f2b5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:15.290906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 115/231] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:14 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:15.297090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc9289d-61e3-4e03-982e-b0269c87f2b5 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:15.311849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43fccce-87b4-4818-a58f-6a999375c3c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:15.312821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43fccce-87b4-4818-a58f-6a999375c3c2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:15.313311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43fccce-87b4-4818-a58f-6a999375c3c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:15.313784 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43fccce-87b4-4818-a58f-6a999375c3c2 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:15.315404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43fccce-87b4-4818-a58f-6a999375c3c2 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:15.317192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43fccce-87b4-4818-a58f-6a999375c3c2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:15.319633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 117/232] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:15 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:15.327049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43fccce-87b4-4818-a58f-6a999375c3c2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:15.454409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc9289d-61e3-4e03-982e-b0269c87f2b5 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:15.476507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:15.506694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:15.521559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dd6a08-31e4-42b4-a43e-1b7dba605056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:15.524168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43fccce-87b4-4818-a58f-6a999375c3c2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:15.524948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dd6a08-31e4-42b4-a43e-1b7dba605056 None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:15.525236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dd6a08-31e4-42b4-a43e-1b7dba605056 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:15.525515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dd6a08-31e4-42b4-a43e-1b7dba605056 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19/users/9c9dd70e89644f408a0d1bad859993f6/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:15.528174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dd6a08-31e4-42b4-a43e-1b7dba605056 None admin] RBAC: Authorizing `identity:create_grant(project_id=dea71e3f98bf423598a93def5ac1cd19, user_id=9c9dd70e89644f408a0d1bad859993f6, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:15.529638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dd6a08-31e4-42b4-a43e-1b7dba605056 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:15.555621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 116/233] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:16:15 2022] PUT /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19/users/9c9dd70e89644f408a0d1bad859993f6/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:15.586696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084835bd-0951-495c-9322-89cd81f01175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:15.587383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084835bd-0951-495c-9322-89cd81f01175 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:15.587663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084835bd-0951-495c-9322-89cd81f01175 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:15.587936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084835bd-0951-495c-9322-89cd81f01175 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19/users/9c9dd70e89644f408a0d1bad859993f6/roles/74db8755b1f04d839260ccd3e36deb66` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:15.591689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084835bd-0951-495c-9322-89cd81f01175 None admin] RBAC: Authorizing `identity:create_grant(project_id=dea71e3f98bf423598a93def5ac1cd19, user_id=9c9dd70e89644f408a0d1bad859993f6, role_id=74db8755b1f04d839260ccd3e36deb66)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:15.593236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084835bd-0951-495c-9322-89cd81f01175 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:15.628394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 118/234] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:16:15 2022] PUT /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19/users/9c9dd70e89644f408a0d1bad859993f6/roles/74db8755b1f04d839260ccd3e36deb66 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:16.691537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6e424-807a-44ec-88a2-8a28dd8107c5 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:16.692054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6e424-807a-44ec-88a2-8a28dd8107c5 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:16.692127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6e424-807a-44ec-88a2-8a28dd8107c5 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:16.693675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 117/235] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:16.699414 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd18eebf-ef18-41d9-a206-d28a0fc76a09 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:16.699845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd18eebf-ef18-41d9-a206-d28a0fc76a09 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:16.700013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd18eebf-ef18-41d9-a206-d28a0fc76a09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:16.725254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-fd18eebf-ef18-41d9-a206-d28a0fc76a09 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:16.726481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-fd18eebf-ef18-41d9-a206-d28a0fc76a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:16.794081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca9e7be-9203-4f56-aa8a-9654f2fbb51e None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:16.794081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca9e7be-9203-4f56-aa8a-9654f2fbb51e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:16.794081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca9e7be-9203-4f56-aa8a-9654f2fbb51e None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:16.794081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 118/236] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:16:16.799831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244d1ee0-d602-4222-9865-9abe9d90e4da None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:16.800354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244d1ee0-d602-4222-9865-9abe9d90e4da None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:16.800837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244d1ee0-d602-4222-9865-9abe9d90e4da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:16.809950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 119/237] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:16.820159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-244d1ee0-d602-4222-9865-9abe9d90e4da None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:16.821617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-244d1ee0-d602-4222-9865-9abe9d90e4da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:16.891246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 119/238] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:17.168717 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd18eebf-ef18-41d9-a206-d28a0fc76a09 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:17.246162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244d1ee0-d602-4222-9865-9abe9d90e4da None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:17.254997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:17.289497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd18eebf-ef18-41d9-a206-d28a0fc76a09 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:17.331365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be28a01-b2b0-4d25-ac56-4be38c9370f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:17.332888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be28a01-b2b0-4d25-ac56-4be38c9370f3 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:17.333177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be28a01-b2b0-4d25-ac56-4be38c9370f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:17.333450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be28a01-b2b0-4d25-ac56-4be38c9370f3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:17.334645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be28a01-b2b0-4d25-ac56-4be38c9370f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:17.336110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be28a01-b2b0-4d25-ac56-4be38c9370f3 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:17.337686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 120/239] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:17.347063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be28a01-b2b0-4d25-ac56-4be38c9370f3 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:17.365054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:17.376096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244d1ee0-d602-4222-9865-9abe9d90e4da None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:17.432596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2687c8-0f0e-489a-abee-aa78899cfae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:17.433603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2687c8-0f0e-489a-abee-aa78899cfae3 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:17.434137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2687c8-0f0e-489a-abee-aa78899cfae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:17.434603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2687c8-0f0e-489a-abee-aa78899cfae3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:17.448708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2687c8-0f0e-489a-abee-aa78899cfae3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-ad2687c8-0f0e-489a-abee-aa78899cfae3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:17.455114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:17.456020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:17.456844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:17.460334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 120/240] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:17 2022] GET /identity/v3/roles/reader => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:17.471476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2687c8-0f0e-489a-abee-aa78899cfae3 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:17.553563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be28a01-b2b0-4d25-ac56-4be38c9370f3 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:17.594723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1065cb8-724d-4518-b938-be486eea00ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:17.595333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1065cb8-724d-4518-b938-be486eea00ce None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:17.595608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1065cb8-724d-4518-b938-be486eea00ce None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:17.595876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1065cb8-724d-4518-b938-be486eea00ce None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:17.596522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1065cb8-724d-4518-b938-be486eea00ce None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:17.597970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1065cb8-724d-4518-b938-be486eea00ce None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:17.623983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 121/241] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:16:17 2022] POST /identity/v3/users => generated 293 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:16:17.626158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2687c8-0f0e-489a-abee-aa78899cfae3 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:17.631796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf326d-0a58-455b-b8ca-a2d5cf239dcf None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:17.632143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf326d-0a58-455b-b8ca-a2d5cf239dcf None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:17.632937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf326d-0a58-455b-b8ca-a2d5cf239dcf None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:17.640069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 122/242] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:17 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:17.646207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de61afd-7281-4780-8d28-239aac417c9e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:17.646588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de61afd-7281-4780-8d28-239aac417c9e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:17.646933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de61afd-7281-4780-8d28-239aac417c9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:17.655039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:17.671879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-5de61afd-7281-4780-8d28-239aac417c9e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:17.673981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5de61afd-7281-4780-8d28-239aac417c9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:17.722984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63398ecb-873f-4b70-975d-b7fd16a7c741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:17.724190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63398ecb-873f-4b70-975d-b7fd16a7c741 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:17.724604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63398ecb-873f-4b70-975d-b7fd16a7c741 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:17.725003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63398ecb-873f-4b70-975d-b7fd16a7c741 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:17.725835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63398ecb-873f-4b70-975d-b7fd16a7c741 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:17.726301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63398ecb-873f-4b70-975d-b7fd16a7c741 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:17.727925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63398ecb-873f-4b70-975d-b7fd16a7c741 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:17.745556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 121/243] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:17 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:17.752012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63398ecb-873f-4b70-975d-b7fd16a7c741 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:17.755178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 123/244] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:17.780970 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:17.867972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63398ecb-873f-4b70-975d-b7fd16a7c741 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:17.914202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8afd6d-848d-4fff-b51e-9336079cc9c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:17.915092 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8afd6d-848d-4fff-b51e-9336079cc9c0 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:17.915533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8afd6d-848d-4fff-b51e-9336079cc9c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:17.915939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8afd6d-848d-4fff-b51e-9336079cc9c0 None admin] PATH_INFO: `/v3/users/e54c9872da02400cb4f4cb9e0a37bece` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:17.935726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8afd6d-848d-4fff-b51e-9336079cc9c0 None admin] RBAC: Authorizing `identity:get_user(user_id=e54c9872da02400cb4f4cb9e0a37bece)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:17.937496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8afd6d-848d-4fff-b51e-9336079cc9c0 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:17.939441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 122/245] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:17 2022] GET /identity/v3/users/e54c9872da02400cb4f4cb9e0a37bece => generated 297 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:17.945210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de61afd-7281-4780-8d28-239aac417c9e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:18.067371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de61afd-7281-4780-8d28-239aac417c9e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:18.102554 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd29ec3-4959-4813-91df-a4df42f927a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:18.103175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd29ec3-4959-4813-91df-a4df42f927a5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:18.103449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd29ec3-4959-4813-91df-a4df42f927a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:18.103717 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd29ec3-4959-4813-91df-a4df42f927a5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:18.106557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:18.129600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8afd6d-848d-4fff-b51e-9336079cc9c0 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:18.179672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd29ec3-4959-4813-91df-a4df42f927a5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:18.180806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd29ec3-4959-4813-91df-a4df42f927a5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:18.217752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:18.248709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8afd6d-848d-4fff-b51e-9336079cc9c0 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:18.294971 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 124/246] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:18.298983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9de9526-a214-4f87-a6ca-164fb4b823d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:18.300366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de9526-a214-4f87-a6ca-164fb4b823d9 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:18.300835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de9526-a214-4f87-a6ca-164fb4b823d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:18.300977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd29ec3-4959-4813-91df-a4df42f927a5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:18.301428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de9526-a214-4f87-a6ca-164fb4b823d9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:18.314801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9de9526-a214-4f87-a6ca-164fb4b823d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-d9de9526-a214-4f87-a6ca-164fb4b823d9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:18.320476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:18.321466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:18.322426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:18.325193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 123/247] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:16:18.331877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9de9526-a214-4f87-a6ca-164fb4b823d9 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:18.338744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:18.415869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd29ec3-4959-4813-91df-a4df42f927a5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:18.453084 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9de9526-a214-4f87-a6ca-164fb4b823d9 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:18.461605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2768f19e-03ba-4e81-af23-fa2a61e4036d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:18.462307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2768f19e-03ba-4e81-af23-fa2a61e4036d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:18.462582 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2768f19e-03ba-4e81-af23-fa2a61e4036d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:18.462851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2768f19e-03ba-4e81-af23-fa2a61e4036d None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:18.464665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2768f19e-03ba-4e81-af23-fa2a61e4036d None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:18.466190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2768f19e-03ba-4e81-af23-fa2a61e4036d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:18.468134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 125/248] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:18 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:18.469927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4fa4f8-d4ff-4a32-88c7-ff8d2b01331a None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:18.469927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4fa4f8-d4ff-4a32-88c7-ff8d2b01331a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:18.470240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4fa4f8-d4ff-4a32-88c7-ff8d2b01331a None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:18.472599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 126/249] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:18.473277 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b8baf0-1d50-4560-b863-c4b4e8f29fe4 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:18.475379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b8baf0-1d50-4560-b863-c4b4e8f29fe4 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:18.475379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b8baf0-1d50-4560-b863-c4b4e8f29fe4 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:18.476049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 127/250] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:18.476614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b8baf0-1d50-4560-b863-c4b4e8f29fe4 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:18.503307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdc6ab4-fe7f-46c8-aee9-3ee78b68adae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:18.503935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc6ab4-fe7f-46c8-aee9-3ee78b68adae None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:18.504214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc6ab4-fe7f-46c8-aee9-3ee78b68adae None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:18.504642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc6ab4-fe7f-46c8-aee9-3ee78b68adae None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:18.505321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdc6ab4-fe7f-46c8-aee9-3ee78b68adae None admin] RBAC: Adding query filter params (name=member) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:18.505623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdc6ab4-fe7f-46c8-aee9-3ee78b68adae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:18.507040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdc6ab4-fe7f-46c8-aee9-3ee78b68adae None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:18.510569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:18.521400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 124/251] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:18.522609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfecca8-c99a-4c4b-abcb-6c91ffac5c1d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:18.522892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfecca8-c99a-4c4b-abcb-6c91ffac5c1d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:18.523162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfecca8-c99a-4c4b-abcb-6c91ffac5c1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:18.543302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-cdfecca8-c99a-4c4b-abcb-6c91ffac5c1d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:18.544458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-cdfecca8-c99a-4c4b-abcb-6c91ffac5c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:18.604943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b8baf0-1d50-4560-b863-c4b4e8f29fe4 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:18.613306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:18.633626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 125/252] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:18.635584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195842db-abe2-4e46-985b-3a11d12a7066 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:18.635868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195842db-abe2-4e46-985b-3a11d12a7066 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:18.636457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195842db-abe2-4e46-985b-3a11d12a7066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:18.662662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7218f08-1e0a-47f6-ae4e-00f530a204c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:18.663655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7218f08-1e0a-47f6-ae4e-00f530a204c4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:18.664129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7218f08-1e0a-47f6-ae4e-00f530a204c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:18.664591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7218f08-1e0a-47f6-ae4e-00f530a204c4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:18.665523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7218f08-1e0a-47f6-ae4e-00f530a204c4 None admin] RBAC: Adding query filter params (role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3, scope.project.id=dea71e3f98bf423598a93def5ac1cd19, user.id=e54c9872da02400cb4f4cb9e0a37bece) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:18.666037 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7218f08-1e0a-47f6-ae4e-00f530a204c4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:18.667820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7218f08-1e0a-47f6-ae4e-00f530a204c4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:18.677410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-195842db-abe2-4e46-985b-3a11d12a7066 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:18.680345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 128/253] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:18 2022] GET /identity/v3/role_assignments?user.id=e54c9872da02400cb4f4cb9e0a37bece&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3 => generated 259 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:18.681854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-195842db-abe2-4e46-985b-3a11d12a7066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:18.684973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7218f08-1e0a-47f6-ae4e-00f530a204c4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:18.845261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 126/254] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:18.909387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7218f08-1e0a-47f6-ae4e-00f530a204c4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:18.961992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ccd1f6-db7b-4a84-a4e9-becb5090db25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:18.962660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ccd1f6-db7b-4a84-a4e9-becb5090db25 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:18.962931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ccd1f6-db7b-4a84-a4e9-becb5090db25 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:18.963199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ccd1f6-db7b-4a84-a4e9-becb5090db25 None admin] PATH_INFO: `/v3/users/8010bf3e93524f76998c82f0cc940491` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:18.964321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ccd1f6-db7b-4a84-a4e9-becb5090db25 None admin] RBAC: Authorizing `identity:get_user(user_id=8010bf3e93524f76998c82f0cc940491)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:18.965768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ccd1f6-db7b-4a84-a4e9-becb5090db25 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:18.967447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 129/255] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:18 2022] GET /identity/v3/users/8010bf3e93524f76998c82f0cc940491 => generated 297 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:18.973254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195842db-abe2-4e46-985b-3a11d12a7066 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:18.973777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ccd1f6-db7b-4a84-a4e9-becb5090db25 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:19.038509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:19.062609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:19.116902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ccd1f6-db7b-4a84-a4e9-becb5090db25 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:19.141079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195842db-abe2-4e46-985b-3a11d12a7066 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:19.178500 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fc1e70-8186-49cf-bca4-6275619b6891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:19.179389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fc1e70-8186-49cf-bca4-6275619b6891 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:19.179809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fc1e70-8186-49cf-bca4-6275619b6891 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:19.180220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fc1e70-8186-49cf-bca4-6275619b6891 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:19.182643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fc1e70-8186-49cf-bca4-6275619b6891 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:19.184280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fc1e70-8186-49cf-bca4-6275619b6891 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:19.186990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 130/256] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:18 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:19.194155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fc1e70-8186-49cf-bca4-6275619b6891 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:19.209741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13328e10-e88b-4e67-af05-2435f6eb6d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:19.210707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13328e10-e88b-4e67-af05-2435f6eb6d0f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:19.211127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13328e10-e88b-4e67-af05-2435f6eb6d0f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:19.211533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13328e10-e88b-4e67-af05-2435f6eb6d0f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:19.234142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13328e10-e88b-4e67-af05-2435f6eb6d0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-13328e10-e88b-4e67-af05-2435f6eb6d0f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:19.239644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:19.240717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:19.241689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:19.249919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 127/257] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:16:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:19.276621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13328e10-e88b-4e67-af05-2435f6eb6d0f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:19.419565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fc1e70-8186-49cf-bca4-6275619b6891 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:19.421374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:19.445142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13328e10-e88b-4e67-af05-2435f6eb6d0f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:19.458287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7b1523-b56a-4383-9e29-8c2163fa5685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:19.458958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7b1523-b56a-4383-9e29-8c2163fa5685 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:19.459233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7b1523-b56a-4383-9e29-8c2163fa5685 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:19.459499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7b1523-b56a-4383-9e29-8c2163fa5685 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19/users/8010bf3e93524f76998c82f0cc940491/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:19.461390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7b1523-b56a-4383-9e29-8c2163fa5685 None admin] RBAC: Authorizing `identity:create_grant(project_id=dea71e3f98bf423598a93def5ac1cd19, user_id=8010bf3e93524f76998c82f0cc940491, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:19.462856 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7b1523-b56a-4383-9e29-8c2163fa5685 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:19.482731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 131/258] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:16:19 2022] PUT /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19/users/8010bf3e93524f76998c82f0cc940491/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:16:19.483596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7b1523-b56a-4383-9e29-8c2163fa5685 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:19.490512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:19.520991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07abfbb-501a-4a96-aa3b-447fd3cb3fc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:19.521633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07abfbb-501a-4a96-aa3b-447fd3cb3fc7 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:19.521907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07abfbb-501a-4a96-aa3b-447fd3cb3fc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:19.522205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07abfbb-501a-4a96-aa3b-447fd3cb3fc7 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:19.522884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07abfbb-501a-4a96-aa3b-447fd3cb3fc7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:19.523181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07abfbb-501a-4a96-aa3b-447fd3cb3fc7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:19.524594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07abfbb-501a-4a96-aa3b-447fd3cb3fc7 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:19.541130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 128/259] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:19.545315 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07abfbb-501a-4a96-aa3b-447fd3cb3fc7 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:19.676529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07abfbb-501a-4a96-aa3b-447fd3cb3fc7 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:19.721195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7b1523-b56a-4383-9e29-8c2163fa5685 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:19.751461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7175a496-e98c-418f-9360-ec2b6f940fbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:19.752125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7175a496-e98c-418f-9360-ec2b6f940fbf None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:19.752400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7175a496-e98c-418f-9360-ec2b6f940fbf None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:19.753856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7175a496-e98c-418f-9360-ec2b6f940fbf None admin] PATH_INFO: `/v3/users/9c9dd70e89644f408a0d1bad859993f6` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:19.759427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7175a496-e98c-418f-9360-ec2b6f940fbf None admin] RBAC: Authorizing `identity:get_user(user_id=9c9dd70e89644f408a0d1bad859993f6)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:19.760936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7175a496-e98c-418f-9360-ec2b6f940fbf None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:19.762664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 129/260] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:19 2022] GET /identity/v3/users/9c9dd70e89644f408a0d1bad859993f6 => generated 283 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:19.768538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7175a496-e98c-418f-9360-ec2b6f940fbf None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:19.799801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49051a22-5d53-4941-b956-3b6192bd8217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:19.800729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49051a22-5d53-4941-b956-3b6192bd8217 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:19.801417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49051a22-5d53-4941-b956-3b6192bd8217 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:19.801699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49051a22-5d53-4941-b956-3b6192bd8217 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:19.830930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49051a22-5d53-4941-b956-3b6192bd8217 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-49051a22-5d53-4941-b956-3b6192bd8217 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:19.839206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:19.840203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:19.840999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:19.842860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 132/261] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 359 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:19.851849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:19.853301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49051a22-5d53-4941-b956-3b6192bd8217 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:19.902073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:19.963313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7175a496-e98c-418f-9360-ec2b6f940fbf None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:20.004979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49051a22-5d53-4941-b956-3b6192bd8217 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:20.038068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3015fbd0-ce70-4a0c-8e09-39293e95df30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:20.039126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3015fbd0-ce70-4a0c-8e09-39293e95df30 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:20.039639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3015fbd0-ce70-4a0c-8e09-39293e95df30 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:20.040223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3015fbd0-ce70-4a0c-8e09-39293e95df30 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:20.045396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaf2e1e-32bf-44d5-9f9c-799ee737b87e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:20.046273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaf2e1e-32bf-44d5-9f9c-799ee737b87e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:20.047128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaf2e1e-32bf-44d5-9f9c-799ee737b87e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:20.047669 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccaf2e1e-32bf-44d5-9f9c-799ee737b87e None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:20.048361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccaf2e1e-32bf-44d5-9f9c-799ee737b87e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:20.048668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccaf2e1e-32bf-44d5-9f9c-799ee737b87e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:20.051039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccaf2e1e-32bf-44d5-9f9c-799ee737b87e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:20.060360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 133/262] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:20.067029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaf2e1e-32bf-44d5-9f9c-799ee737b87e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:20.167355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:20.185243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3015fbd0-ce70-4a0c-8e09-39293e95df30 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:20.186144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccaf2e1e-32bf-44d5-9f9c-799ee737b87e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:20.187596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3015fbd0-ce70-4a0c-8e09-39293e95df30 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:20.229216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8725ffcc-aab6-41da-ba25-8dd229188dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:20.229873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8725ffcc-aab6-41da-ba25-8dd229188dff None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:20.230176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8725ffcc-aab6-41da-ba25-8dd229188dff None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:20.230459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8725ffcc-aab6-41da-ba25-8dd229188dff None admin] PATH_INFO: `/v3/users/9c9dd70e89644f408a0d1bad859993f6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:20.231855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8725ffcc-aab6-41da-ba25-8dd229188dff None admin] RBAC: Authorizing `identity:get_user(user_id=9c9dd70e89644f408a0d1bad859993f6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:20.233644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8725ffcc-aab6-41da-ba25-8dd229188dff None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:20.236309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 134/263] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:20 2022] GET /identity/v3/users/9c9dd70e89644f408a0d1bad859993f6 => generated 283 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:20.242303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8725ffcc-aab6-41da-ba25-8dd229188dff None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:20.305495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:20.356882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8725ffcc-aab6-41da-ba25-8dd229188dff None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:20.395526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fddc09-dec6-4ef9-bb66-2bb03daa645c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:20.396161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fddc09-dec6-4ef9-bb66-2bb03daa645c None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:20.396445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fddc09-dec6-4ef9-bb66-2bb03daa645c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:20.396721 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fddc09-dec6-4ef9-bb66-2bb03daa645c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:20.422506 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:20.432088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 130/264] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 664 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:20.443180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3015fbd0-ce70-4a0c-8e09-39293e95df30 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:20.473616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fddc09-dec6-4ef9-bb66-2bb03daa645c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:20.475323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fddc09-dec6-4ef9-bb66-2bb03daa645c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:20.582948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 135/265] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:20.588527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fddc09-dec6-4ef9-bb66-2bb03daa645c None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:20.660107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3015fbd0-ce70-4a0c-8e09-39293e95df30 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:20.678832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:20.694569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fddc09-dec6-4ef9-bb66-2bb03daa645c None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:20.731271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96744fab-f167-4aa2-a9ca-d00815fdeb25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:20.731931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96744fab-f167-4aa2-a9ca-d00815fdeb25 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:20.732213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96744fab-f167-4aa2-a9ca-d00815fdeb25 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:20.732484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96744fab-f167-4aa2-a9ca-d00815fdeb25 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:20.733749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96744fab-f167-4aa2-a9ca-d00815fdeb25 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:20.736084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96744fab-f167-4aa2-a9ca-d00815fdeb25 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:20.738072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 136/266] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:20 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:20.744405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96744fab-f167-4aa2-a9ca-d00815fdeb25 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:20.750925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a765f0df-40dd-4d5c-8c1b-5ae2e7dac539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:20.751592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a765f0df-40dd-4d5c-8c1b-5ae2e7dac539 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:20.751871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a765f0df-40dd-4d5c-8c1b-5ae2e7dac539 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:20.752142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a765f0df-40dd-4d5c-8c1b-5ae2e7dac539 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:20.753393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a765f0df-40dd-4d5c-8c1b-5ae2e7dac539 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:20.754924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a765f0df-40dd-4d5c-8c1b-5ae2e7dac539 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:20.756878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 131/267] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:20 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:20.762668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a765f0df-40dd-4d5c-8c1b-5ae2e7dac539 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:20.804152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:20.915684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a765f0df-40dd-4d5c-8c1b-5ae2e7dac539 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:20.921593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96744fab-f167-4aa2-a9ca-d00815fdeb25 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:20.957789 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f25859-f7a3-49bc-a478-7be2320b6130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:20.958581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f25859-f7a3-49bc-a478-7be2320b6130 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:20.958870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f25859-f7a3-49bc-a478-7be2320b6130 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:20.959141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f25859-f7a3-49bc-a478-7be2320b6130 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:20.960063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f25859-f7a3-49bc-a478-7be2320b6130 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=dea71e3f98bf423598a93def5ac1cd19, user.id=9c9dd70e89644f408a0d1bad859993f6) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:20.960379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f25859-f7a3-49bc-a478-7be2320b6130 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:20.963543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f25859-f7a3-49bc-a478-7be2320b6130 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:20.970001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:20.972244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 132/268] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:20 2022] GET /identity/v3/role_assignments?user.id=9c9dd70e89644f408a0d1bad859993f6&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=22819414415942d399db534708fe7103 => generated 608 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:21.008882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c09490-96be-4dad-a8ba-1b7398c81bd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:21.009522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c09490-96be-4dad-a8ba-1b7398c81bd0 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:21.009803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c09490-96be-4dad-a8ba-1b7398c81bd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:21.010074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c09490-96be-4dad-a8ba-1b7398c81bd0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:21.010852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c09490-96be-4dad-a8ba-1b7398c81bd0 None admin] RBAC: Adding query filter params (role.id=74db8755b1f04d839260ccd3e36deb66, scope.project.id=dea71e3f98bf423598a93def5ac1cd19, user.id=9c9dd70e89644f408a0d1bad859993f6) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:21.011161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c09490-96be-4dad-a8ba-1b7398c81bd0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:21.012634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c09490-96be-4dad-a8ba-1b7398c81bd0 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:21.023495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 137/269] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:20 2022] GET /identity/v3/role_assignments?user.id=9c9dd70e89644f408a0d1bad859993f6&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=74db8755b1f04d839260ccd3e36deb66 => generated 608 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:21.328744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467e16c0-9c6e-4cdf-8376-da50e361eb38 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:21.328920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467e16c0-9c6e-4cdf-8376-da50e361eb38 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:21.329187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467e16c0-9c6e-4cdf-8376-da50e361eb38 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:21.331055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 133/270] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:21.338537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1e48d7-8075-4f49-a6d4-bf297dd3e38e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:21.338852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1e48d7-8075-4f49-a6d4-bf297dd3e38e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:21.339079 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1e48d7-8075-4f49-a6d4-bf297dd3e38e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:21.364715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0b1e48d7-8075-4f49-a6d4-bf297dd3e38e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:21.366066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0b1e48d7-8075-4f49-a6d4-bf297dd3e38e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:21.454180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 138/271] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:21.461973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8ace43-bec0-4fa1-82d3-a46061b6f036 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:21.462385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8ace43-bec0-4fa1-82d3-a46061b6f036 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:21.462845 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8ace43-bec0-4fa1-82d3-a46061b6f036 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:21.464301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 134/272] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:21.470142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7737d794-827f-45ef-ab60-bb7270046b40 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:21.470408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7737d794-827f-45ef-ab60-bb7270046b40 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:21.470679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7737d794-827f-45ef-ab60-bb7270046b40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:21.492759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7737d794-827f-45ef-ab60-bb7270046b40 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:21.493954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7737d794-827f-45ef-ab60-bb7270046b40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:21.566131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 139/273] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:21.760321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8ace43-bec0-4fa1-82d3-a46061b6f036 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:21.845013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:21.855794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7737d794-827f-45ef-ab60-bb7270046b40 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:21.872202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8ace43-bec0-4fa1-82d3-a46061b6f036 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:21.904217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:21.907406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adfd7ff-2b2e-4343-ba20-b6cc72e18bbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:21.908051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adfd7ff-2b2e-4343-ba20-b6cc72e18bbf None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:21.908381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adfd7ff-2b2e-4343-ba20-b6cc72e18bbf None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:21.908673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adfd7ff-2b2e-4343-ba20-b6cc72e18bbf None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:21.909710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adfd7ff-2b2e-4343-ba20-b6cc72e18bbf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:21.911197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adfd7ff-2b2e-4343-ba20-b6cc72e18bbf None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:21.914256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 135/274] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:21.914256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3e2dd-767b-429e-9499-d26ed88ff85a None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:21.914256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3e2dd-767b-429e-9499-d26ed88ff85a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:21.914524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3e2dd-767b-429e-9499-d26ed88ff85a None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:21.919466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 136/275] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:21 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:21.919466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a3e2dd-767b-429e-9499-d26ed88ff85a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:21.964476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7737d794-827f-45ef-ab60-bb7270046b40 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:21.999720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505b932d-e401-4e7d-a294-de6e9245ff4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:22.000739 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505b932d-e401-4e7d-a294-de6e9245ff4d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:22.001083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505b932d-e401-4e7d-a294-de6e9245ff4d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:22.001410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505b932d-e401-4e7d-a294-de6e9245ff4d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:22.011983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505b932d-e401-4e7d-a294-de6e9245ff4d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-505b932d-e401-4e7d-a294-de6e9245ff4d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:22.016943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:22.018025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 140/276] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:22.023666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58980bb7-f37a-408b-a808-09a971d7b741 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:22.024593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58980bb7-f37a-408b-a808-09a971d7b741 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:22.024593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58980bb7-f37a-408b-a808-09a971d7b741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:22.038734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-58980bb7-f37a-408b-a808-09a971d7b741 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:22.040047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-58980bb7-f37a-408b-a808-09a971d7b741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:22.043548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a3e2dd-767b-429e-9499-d26ed88ff85a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:22.081644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b31adc-adcf-44cc-b807-8b32f5308134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:22.082288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b31adc-adcf-44cc-b807-8b32f5308134 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:22.082564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b31adc-adcf-44cc-b807-8b32f5308134 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:22.082832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b31adc-adcf-44cc-b807-8b32f5308134 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:22.083475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b31adc-adcf-44cc-b807-8b32f5308134 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:22.084923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b31adc-adcf-44cc-b807-8b32f5308134 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:22.108909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 137/277] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:16:21 2022] POST /identity/v3/users => generated 293 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:16:22.109892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b31adc-adcf-44cc-b807-8b32f5308134 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:22.111416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 141/278] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:22.117691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:22.235269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b31adc-adcf-44cc-b807-8b32f5308134 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:22.272528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7c5739-0f6a-49b5-9224-92f9c9d7805f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:22.273168 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7c5739-0f6a-49b5-9224-92f9c9d7805f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:22.273447 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7c5739-0f6a-49b5-9224-92f9c9d7805f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:22.273732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7c5739-0f6a-49b5-9224-92f9c9d7805f None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:22.274565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7c5739-0f6a-49b5-9224-92f9c9d7805f None admin] RBAC: Adding query filter params (name=member) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:22.275177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7c5739-0f6a-49b5-9224-92f9c9d7805f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:22.276676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7c5739-0f6a-49b5-9224-92f9c9d7805f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:22.287236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 138/279] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:22.293134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58980bb7-f37a-408b-a808-09a971d7b741 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:22.335189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:22.381466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7c5739-0f6a-49b5-9224-92f9c9d7805f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:22.411580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58980bb7-f37a-408b-a808-09a971d7b741 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:22.453825 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443aaa00-4413-44b6-ab11-9ea6136333a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:22.454544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443aaa00-4413-44b6-ab11-9ea6136333a6 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:22.455085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443aaa00-4413-44b6-ab11-9ea6136333a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:22.455357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443aaa00-4413-44b6-ab11-9ea6136333a6 None admin] PATH_INFO: `/v3/users/9dfc400edaaf43cb9c7cb54740656bc8` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:22.469272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:22.476010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443aaa00-4413-44b6-ab11-9ea6136333a6 None admin] RBAC: Authorizing `identity:get_user(user_id=9dfc400edaaf43cb9c7cb54740656bc8)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:22.477544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443aaa00-4413-44b6-ab11-9ea6136333a6 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:22.480722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 142/280] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:22 2022] GET /identity/v3/users/9dfc400edaaf43cb9c7cb54740656bc8 => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:22.488495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443aaa00-4413-44b6-ab11-9ea6136333a6 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:22.527316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7c5739-0f6a-49b5-9224-92f9c9d7805f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:22.570426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8465d2-6436-4ea1-a287-c7a831ce862b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:22.571083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8465d2-6436-4ea1-a287-c7a831ce862b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:22.571355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8465d2-6436-4ea1-a287-c7a831ce862b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:22.571784 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8465d2-6436-4ea1-a287-c7a831ce862b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:22.585198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8465d2-6436-4ea1-a287-c7a831ce862b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-ac8465d2-6436-4ea1-a287-c7a831ce862b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:22.590500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:22.591343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:22.593055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 139/281] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:16:22.597545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8465d2-6436-4ea1-a287-c7a831ce862b None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:22.615071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:22.626799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443aaa00-4413-44b6-ab11-9ea6136333a6 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:22.673303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e1dd53-3a49-44e5-ac17-eba08392aed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:22.673996 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e1dd53-3a49-44e5-ac17-eba08392aed3 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:22.674448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e1dd53-3a49-44e5-ac17-eba08392aed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:22.674719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e1dd53-3a49-44e5-ac17-eba08392aed3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:22.675446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e1dd53-3a49-44e5-ac17-eba08392aed3 None admin] RBAC: Adding query filter params (role.id=7a7e816668a7493bad0752baca37934c, user.id=9dfc400edaaf43cb9c7cb54740656bc8, scope.system=all) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:22.675751 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e1dd53-3a49-44e5-ac17-eba08392aed3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:22.678319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e1dd53-3a49-44e5-ac17-eba08392aed3 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:22.688282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 143/282] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Sep 13 15:16:22 2022] GET /identity/v3/role_assignments?user.id=9dfc400edaaf43cb9c7cb54740656bc8&scope.system=all&role.id=7a7e816668a7493bad0752baca37934c => generated 226 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:22.694137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed5cb11-fb49-499b-b84b-788e00ee94e0 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:22.694137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed5cb11-fb49-499b-b84b-788e00ee94e0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:22.694137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed5cb11-fb49-499b-b84b-788e00ee94e0 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:22.694137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 144/283] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:22.698156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3363f8-9546-4d7e-a0d8-5cdc9b4c29f3 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:22.698445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3363f8-9546-4d7e-a0d8-5cdc9b4c29f3 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:22.698711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3363f8-9546-4d7e-a0d8-5cdc9b4c29f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:22.721470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-2a3363f8-9546-4d7e-a0d8-5cdc9b4c29f3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:22.722792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2a3363f8-9546-4d7e-a0d8-5cdc9b4c29f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:22.751896 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8465d2-6436-4ea1-a287-c7a831ce862b None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:22.814747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:22.815485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 145/284] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:22.839151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f6e106-ca2e-452d-8320-28f21b35fb17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:22.839794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f6e106-ca2e-452d-8320-28f21b35fb17 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:22.840066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f6e106-ca2e-452d-8320-28f21b35fb17 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:22.840333 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f6e106-ca2e-452d-8320-28f21b35fb17 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:22.841061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f6e106-ca2e-452d-8320-28f21b35fb17 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:22.841425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f6e106-ca2e-452d-8320-28f21b35fb17 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:22.843048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f6e106-ca2e-452d-8320-28f21b35fb17 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:22.862513 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 140/285] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:22.869685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3363f8-9546-4d7e-a0d8-5cdc9b4c29f3 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:22.974141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:22.991382 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3363f8-9546-4d7e-a0d8-5cdc9b4c29f3 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:23.031155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6ce031-4578-4ee1-a4fa-64ee66d5a7a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:23.032718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ce031-4578-4ee1-a4fa-64ee66d5a7a2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:23.033184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ce031-4578-4ee1-a4fa-64ee66d5a7a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:23.033652 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ce031-4578-4ee1-a4fa-64ee66d5a7a2 None admin] PATH_INFO: `/v3/users/e54c9872da02400cb4f4cb9e0a37bece` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:23.035016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6ce031-4578-4ee1-a4fa-64ee66d5a7a2 None admin] RBAC: Authorizing `identity:get_user(user_id=e54c9872da02400cb4f4cb9e0a37bece)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:23.036645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6ce031-4578-4ee1-a4fa-64ee66d5a7a2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:23.038909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 146/286] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:22 2022] GET /identity/v3/users/e54c9872da02400cb4f4cb9e0a37bece => generated 297 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:23.046136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f6e106-ca2e-452d-8320-28f21b35fb17 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:23.146917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6ce031-4578-4ee1-a4fa-64ee66d5a7a2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:23.263944 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6ce031-4578-4ee1-a4fa-64ee66d5a7a2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:23.291179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f6e106-ca2e-452d-8320-28f21b35fb17 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:23.304573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de9e86c-b04a-4eb6-b111-43585bc3a997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:23.305641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de9e86c-b04a-4eb6-b111-43585bc3a997 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:23.306343 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de9e86c-b04a-4eb6-b111-43585bc3a997 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:23.306803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de9e86c-b04a-4eb6-b111-43585bc3a997 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:23.322458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de9e86c-b04a-4eb6-b111-43585bc3a997 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-0de9e86c-b04a-4eb6-b111-43585bc3a997 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:23.332016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:23.333545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:23.334424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:23.338089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 147/287] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:23.347374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de9e86c-b04a-4eb6-b111-43585bc3a997 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:23.364429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8507435-fcda-42d5-a980-74f3d7e02a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:23.365082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8507435-fcda-42d5-a980-74f3d7e02a1d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:23.365359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8507435-fcda-42d5-a980-74f3d7e02a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:23.365628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8507435-fcda-42d5-a980-74f3d7e02a1d None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:23.368444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8507435-fcda-42d5-a980-74f3d7e02a1d None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:23.369929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8507435-fcda-42d5-a980-74f3d7e02a1d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:23.371629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 141/288] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:23 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:23.378205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8507435-fcda-42d5-a980-74f3d7e02a1d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:23.378437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:23.446509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:23.490894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8507435-fcda-42d5-a980-74f3d7e02a1d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:23.526193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad6aa31-bbc4-4df0-ab08-380d0fe51739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:23.526922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad6aa31-bbc4-4df0-ab08-380d0fe51739 None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:23.528705 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad6aa31-bbc4-4df0-ab08-380d0fe51739 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:23.528992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad6aa31-bbc4-4df0-ab08-380d0fe51739 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19/users/e54c9872da02400cb4f4cb9e0a37bece/roles/0ff8d8f3f5ae426197cdffd9ea41a4e3` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:23.531083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad6aa31-bbc4-4df0-ab08-380d0fe51739 None admin] RBAC: Authorizing `identity:create_grant(project_id=dea71e3f98bf423598a93def5ac1cd19, user_id=e54c9872da02400cb4f4cb9e0a37bece, role_id=0ff8d8f3f5ae426197cdffd9ea41a4e3)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:23.532641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad6aa31-bbc4-4df0-ab08-380d0fe51739 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:23.553992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 142/289] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:16:23 2022] PUT /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19/users/e54c9872da02400cb4f4cb9e0a37bece/roles/0ff8d8f3f5ae426197cdffd9ea41a4e3 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:23.611315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de9e86c-b04a-4eb6-b111-43585bc3a997 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:23.663709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b0fbe2-b9bf-4a48-9b08-c259ee7bb0cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:23.664355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0fbe2-b9bf-4a48-9b08-c259ee7bb0cd None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:23.664633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0fbe2-b9bf-4a48-9b08-c259ee7bb0cd None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:23.664904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b0fbe2-b9bf-4a48-9b08-c259ee7bb0cd None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:23.665594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b0fbe2-b9bf-4a48-9b08-c259ee7bb0cd None admin] RBAC: Adding query filter params (name=member) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:23.665900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b0fbe2-b9bf-4a48-9b08-c259ee7bb0cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:23.667548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b0fbe2-b9bf-4a48-9b08-c259ee7bb0cd None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:23.683417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 148/290] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:23.691756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad6aa31-bbc4-4df0-ab08-380d0fe51739 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:23.823152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad6aa31-bbc4-4df0-ab08-380d0fe51739 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:23.843352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:23.864035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5c2c9d-bbf6-47b2-9903-b0161fc3084d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:23.864690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5c2c9d-bbf6-47b2-9903-b0161fc3084d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:23.864964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5c2c9d-bbf6-47b2-9903-b0161fc3084d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:23.865234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5c2c9d-bbf6-47b2-9903-b0161fc3084d None admin] PATH_INFO: `/v3/users/8010bf3e93524f76998c82f0cc940491` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:23.866409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5c2c9d-bbf6-47b2-9903-b0161fc3084d None admin] RBAC: Authorizing `identity:get_user(user_id=8010bf3e93524f76998c82f0cc940491)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:23.868045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5c2c9d-bbf6-47b2-9903-b0161fc3084d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:23.869985 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 143/291] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:23 2022] GET /identity/v3/users/8010bf3e93524f76998c82f0cc940491 => generated 297 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:23.877603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b0fbe2-b9bf-4a48-9b08-c259ee7bb0cd None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:23.972245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:24.007879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b0fbe2-b9bf-4a48-9b08-c259ee7bb0cd None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:24.054043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8f088e-e1a9-4fc1-aee4-26bb92793048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:24.055481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8f088e-e1a9-4fc1-aee4-26bb92793048 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:24.055965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8f088e-e1a9-4fc1-aee4-26bb92793048 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:24.056442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8f088e-e1a9-4fc1-aee4-26bb92793048 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:24.154029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8f088e-e1a9-4fc1-aee4-26bb92793048 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:24.156011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8f088e-e1a9-4fc1-aee4-26bb92793048 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:24.244598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:24.274335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 149/292] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:24.279922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5c2c9d-bbf6-47b2-9903-b0161fc3084d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:24.415908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5c2c9d-bbf6-47b2-9903-b0161fc3084d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:24.451155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a579f12-e963-4d27-b015-862e6c2a1e1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:24.451798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a579f12-e963-4d27-b015-862e6c2a1e1f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:24.452077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a579f12-e963-4d27-b015-862e6c2a1e1f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:24.452554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a579f12-e963-4d27-b015-862e6c2a1e1f None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:24.453694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a579f12-e963-4d27-b015-862e6c2a1e1f None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:24.455188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a579f12-e963-4d27-b015-862e6c2a1e1f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:24.456822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 144/293] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:24 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:24.462904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8f088e-e1a9-4fc1-aee4-26bb92793048 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:24.594490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8f088e-e1a9-4fc1-aee4-26bb92793048 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:24.646872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25797c66-47e2-473e-bd37-60425b6fa8e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:24.647513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25797c66-47e2-473e-bd37-60425b6fa8e2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:24.647793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25797c66-47e2-473e-bd37-60425b6fa8e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:24.648062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25797c66-47e2-473e-bd37-60425b6fa8e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:24.648798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25797c66-47e2-473e-bd37-60425b6fa8e2 None admin] RBAC: Adding query filter params (role.id=7a7e816668a7493bad0752baca37934c, scope.project.id=dea71e3f98bf423598a93def5ac1cd19, user.id=8010bf3e93524f76998c82f0cc940491) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:24.649104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25797c66-47e2-473e-bd37-60425b6fa8e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:24.650565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25797c66-47e2-473e-bd37-60425b6fa8e2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:24.663328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 150/294] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:24 2022] GET /identity/v3/role_assignments?user.id=8010bf3e93524f76998c82f0cc940491&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=7a7e816668a7493bad0752baca37934c => generated 608 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:25.111938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079d6582-9d53-4e97-b12a-02b22c404257 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:25.112357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079d6582-9d53-4e97-b12a-02b22c404257 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:25.112456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079d6582-9d53-4e97-b12a-02b22c404257 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:25.114450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 145/295] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:25.119936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36929d2e-0ecb-4337-81a7-23b5455ea222 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:25.120370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36929d2e-0ecb-4337-81a7-23b5455ea222 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:25.120462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36929d2e-0ecb-4337-81a7-23b5455ea222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:25.142521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-36929d2e-0ecb-4337-81a7-23b5455ea222 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:25.144112 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-36929d2e-0ecb-4337-81a7-23b5455ea222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:25.184331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:25.226187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 151/296] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:25.274090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ecc52-9782-4f43-9f49-45b7fcb65da7 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:25.274460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ecc52-9782-4f43-9f49-45b7fcb65da7 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:25.274650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658ecc52-9782-4f43-9f49-45b7fcb65da7 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:25.276158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 146/297] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:25.282545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874ac696-bb40-4917-845b-16c2229c9f30 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:25.283168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874ac696-bb40-4917-845b-16c2229c9f30 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:25.283655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874ac696-bb40-4917-845b-16c2229c9f30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:25.304488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-874ac696-bb40-4917-845b-16c2229c9f30 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:25.305983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-874ac696-bb40-4917-845b-16c2229c9f30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:25.380363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 152/298] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:25.498959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658ecc52-9782-4f43-9f49-45b7fcb65da7 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:25.558701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:25.586005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6bd391-5038-4c63-a462-a0e5c1a9cb36 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:25.586332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6bd391-5038-4c63-a462-a0e5c1a9cb36 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:25.586589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6bd391-5038-4c63-a462-a0e5c1a9cb36 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:25.588176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 153/299] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:16:25.594041 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daff906-a289-41f7-b1f5-b445e9116254 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:25.594349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daff906-a289-41f7-b1f5-b445e9116254 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:25.594622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daff906-a289-41f7-b1f5-b445e9116254 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:25.611346 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658ecc52-9782-4f43-9f49-45b7fcb65da7 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:25.614045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7daff906-a289-41f7-b1f5-b445e9116254 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:25.615257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7daff906-a289-41f7-b1f5-b445e9116254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:25.653342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecdcb9f-6ac8-45e2-8cb2-bbef430405cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:25.654308 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecdcb9f-6ac8-45e2-8cb2-bbef430405cb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:25.654866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecdcb9f-6ac8-45e2-8cb2-bbef430405cb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:25.655743 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecdcb9f-6ac8-45e2-8cb2-bbef430405cb None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:25.667909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecdcb9f-6ac8-45e2-8cb2-bbef430405cb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-2ecdcb9f-6ac8-45e2-8cb2-bbef430405cb None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:25.674022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:25.674931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:25.675706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:25.679974 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 147/300] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:25.680719 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecdcb9f-6ac8-45e2-8cb2-bbef430405cb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:25.697900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 154/301] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:25.703280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7daff906-a289-41f7-b1f5-b445e9116254 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:25.739017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:25.786526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecdcb9f-6ac8-45e2-8cb2-bbef430405cb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:25.800170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:25.822575 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe019d7-cd27-48c2-ba9b-61169484cdc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:25.823241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe019d7-cd27-48c2-ba9b-61169484cdc8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:25.823512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe019d7-cd27-48c2-ba9b-61169484cdc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:25.823777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe019d7-cd27-48c2-ba9b-61169484cdc8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:25.824922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe019d7-cd27-48c2-ba9b-61169484cdc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:25.826416 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe019d7-cd27-48c2-ba9b-61169484cdc8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:25.832793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 148/302] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:25.834859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe019d7-cd27-48c2-ba9b-61169484cdc8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:25.837429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7daff906-a289-41f7-b1f5-b445e9116254 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:25.880712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a35300-01b1-480e-8d9d-84befabf018f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:25.882135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a35300-01b1-480e-8d9d-84befabf018f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:25.882135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a35300-01b1-480e-8d9d-84befabf018f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:25.882135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a35300-01b1-480e-8d9d-84befabf018f None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:25.882566 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a35300-01b1-480e-8d9d-84befabf018f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:25.882908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a35300-01b1-480e-8d9d-84befabf018f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:25.884339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a35300-01b1-480e-8d9d-84befabf018f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:25.899691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 155/303] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:25.901023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a35300-01b1-480e-8d9d-84befabf018f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:25.971319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe019d7-cd27-48c2-ba9b-61169484cdc8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:26.006968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:26.011023 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3c1355-5f99-4053-9ba9-90c07d4bc921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:26.011903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3c1355-5f99-4053-9ba9-90c07d4bc921 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:26.012183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3c1355-5f99-4053-9ba9-90c07d4bc921 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:26.012450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3c1355-5f99-4053-9ba9-90c07d4bc921 None admin] PATH_INFO: `/v3/groups` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:26.013283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3c1355-5f99-4053-9ba9-90c07d4bc921 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:26.015483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3c1355-5f99-4053-9ba9-90c07d4bc921 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:26.016811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a35300-01b1-480e-8d9d-84befabf018f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:26.042250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 149/304] 10.222.0.16 () {60 vars in 1134 bytes} [Tue Sep 13 15:16:25 2022] POST /identity/v3/groups => generated 229 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:16:26.042250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3c1355-5f99-4053-9ba9-90c07d4bc921 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:26.052044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddea0ce-d748-433a-915d-ac0ceda4c81f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:26.052869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddea0ce-d748-433a-915d-ac0ceda4c81f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:26.053274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddea0ce-d748-433a-915d-ac0ceda4c81f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:26.053713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddea0ce-d748-433a-915d-ac0ceda4c81f None admin] PATH_INFO: `/v3/users/f223de0989bd4b978ec0f8ec8e64f728` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:26.070917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddea0ce-d748-433a-915d-ac0ceda4c81f None admin] RBAC: Authorizing `identity:get_user(user_id=f223de0989bd4b978ec0f8ec8e64f728)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:26.072621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddea0ce-d748-433a-915d-ac0ceda4c81f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:26.079465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 156/305] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:25 2022] GET /identity/v3/users/f223de0989bd4b978ec0f8ec8e64f728 => generated 293 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:26.081204 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddea0ce-d748-433a-915d-ac0ceda4c81f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:26.098907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:26.186487 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3c1355-5f99-4053-9ba9-90c07d4bc921 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:26.201890 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddea0ce-d748-433a-915d-ac0ceda4c81f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:26.235832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2086e26-0560-41e0-b0c2-33cffcfcf8d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:26.236925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2086e26-0560-41e0-b0c2-33cffcfcf8d1 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:26.237559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2086e26-0560-41e0-b0c2-33cffcfcf8d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:26.238330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2086e26-0560-41e0-b0c2-33cffcfcf8d1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:26.244504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8066c96d-9c18-4b21-9206-1929c297daf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:26.245186 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8066c96d-9c18-4b21-9206-1929c297daf8 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:26.245464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8066c96d-9c18-4b21-9206-1929c297daf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:26.245718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8066c96d-9c18-4b21-9206-1929c297daf8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:26.246518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8066c96d-9c18-4b21-9206-1929c297daf8 None admin] RBAC: Adding query filter params (role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3, user.id=f223de0989bd4b978ec0f8ec8e64f728, scope.system=all) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:26.246839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8066c96d-9c18-4b21-9206-1929c297daf8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:26.248365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8066c96d-9c18-4b21-9206-1929c297daf8 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:26.254763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2086e26-0560-41e0-b0c2-33cffcfcf8d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:26.255063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 157/306] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Sep 13 15:16:26 2022] GET /identity/v3/role_assignments?user.id=f223de0989bd4b978ec0f8ec8e64f728&scope.system=all&role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3 => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:26.259131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c50e697-8c16-4d46-a6f7-5d5dee2fb035 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:26.259422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c50e697-8c16-4d46-a6f7-5d5dee2fb035 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:26.259687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c50e697-8c16-4d46-a6f7-5d5dee2fb035 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:26.261415 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 158/307] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-c2086e26-0560-41e0-b0c2-33cffcfcf8d1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:26.262047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:26.262917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:26.263710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:26.266667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 150/308] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:26.267502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4121dc00-9c97-4a3e-9339-ae96e40eafeb None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:26.267800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4121dc00-9c97-4a3e-9339-ae96e40eafeb None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:26.268064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4121dc00-9c97-4a3e-9339-ae96e40eafeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:26.273713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2086e26-0560-41e0-b0c2-33cffcfcf8d1 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:26.289189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4121dc00-9c97-4a3e-9339-ae96e40eafeb None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:26.290525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4121dc00-9c97-4a3e-9339-ae96e40eafeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:26.302198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:26.389296 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 159/309] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:26.413880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2086e26-0560-41e0-b0c2-33cffcfcf8d1 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:26.422505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:26.452631 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e510b-7296-4975-88b2-5f225c2f1aed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:26.453352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e510b-7296-4975-88b2-5f225c2f1aed None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:26.453684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e510b-7296-4975-88b2-5f225c2f1aed None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:26.453962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e510b-7296-4975-88b2-5f225c2f1aed None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:26.454694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1e510b-7296-4975-88b2-5f225c2f1aed None admin] RBAC: Adding query filter params (name=member) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:26.454997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1e510b-7296-4975-88b2-5f225c2f1aed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:26.456404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1e510b-7296-4975-88b2-5f225c2f1aed None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:26.466286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 151/310] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:26.475320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4121dc00-9c97-4a3e-9339-ae96e40eafeb None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:26.577546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4121dc00-9c97-4a3e-9339-ae96e40eafeb None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:26.612028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f051166e-6e58-4a3a-b718-def8c1e102af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:26.612679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f051166e-6e58-4a3a-b718-def8c1e102af None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:26.612948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f051166e-6e58-4a3a-b718-def8c1e102af None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:26.613215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f051166e-6e58-4a3a-b718-def8c1e102af None admin] PATH_INFO: `/v3/users/9dfc400edaaf43cb9c7cb54740656bc8` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:26.614771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f051166e-6e58-4a3a-b718-def8c1e102af None admin] RBAC: Authorizing `identity:get_user(user_id=9dfc400edaaf43cb9c7cb54740656bc8)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:26.616255 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f051166e-6e58-4a3a-b718-def8c1e102af None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:26.618151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 160/311] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:26 2022] GET /identity/v3/users/9dfc400edaaf43cb9c7cb54740656bc8 => generated 293 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:26.623841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e510b-7296-4975-88b2-5f225c2f1aed None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:26.696030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f051166e-6e58-4a3a-b718-def8c1e102af None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:26.711969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:26.747032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e510b-7296-4975-88b2-5f225c2f1aed None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:26.795623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491c2faf-f26a-4ba7-8063-1c8902d24d28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:26.796990 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491c2faf-f26a-4ba7-8063-1c8902d24d28 None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:26.797459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491c2faf-f26a-4ba7-8063-1c8902d24d28 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:26.798465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491c2faf-f26a-4ba7-8063-1c8902d24d28 None admin] PATH_INFO: `/v3/system/users/9dfc400edaaf43cb9c7cb54740656bc8/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:26.800603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491c2faf-f26a-4ba7-8063-1c8902d24d28 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=9dfc400edaaf43cb9c7cb54740656bc8, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:26.802915 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491c2faf-f26a-4ba7-8063-1c8902d24d28 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:26.810945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f051166e-6e58-4a3a-b718-def8c1e102af None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:26.816906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 152/312] 10.222.0.16 () {58 vars in 1334 bytes} [Tue Sep 13 15:16:26 2022] PUT /identity/v3/system/users/9dfc400edaaf43cb9c7cb54740656bc8/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:26.854869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1282fa9-3672-43f8-b5cb-b6523ceb9d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:26.855684 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1282fa9-3672-43f8-b5cb-b6523ceb9d2a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:26.855977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1282fa9-3672-43f8-b5cb-b6523ceb9d2a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:26.856310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1282fa9-3672-43f8-b5cb-b6523ceb9d2a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:26.869605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1282fa9-3672-43f8-b5cb-b6523ceb9d2a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-b1282fa9-3672-43f8-b5cb-b6523ceb9d2a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:26.875586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:26.876510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:26.877319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 161/313] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:26 2022] GET /identity/v3/roles/reader => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:16:26.883231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491c2faf-f26a-4ba7-8063-1c8902d24d28 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:26.914939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:27.007453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491c2faf-f26a-4ba7-8063-1c8902d24d28 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:27.048676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7273b6d6-24c9-4798-bba4-6181dfdfdfe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:27.049668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7273b6d6-24c9-4798-bba4-6181dfdfdfe4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:27.050178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7273b6d6-24c9-4798-bba4-6181dfdfdfe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:27.050635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7273b6d6-24c9-4798-bba4-6181dfdfdfe4 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:27.051503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7273b6d6-24c9-4798-bba4-6181dfdfdfe4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:27.051987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7273b6d6-24c9-4798-bba4-6181dfdfdfe4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:27.053599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7273b6d6-24c9-4798-bba4-6181dfdfdfe4 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:27.063066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 153/314] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:26 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:27.068994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1282fa9-3672-43f8-b5cb-b6523ceb9d2a None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:27.145191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:27.178386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1282fa9-3672-43f8-b5cb-b6523ceb9d2a None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:27.213639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d539775-8971-4a45-b382-af134779e114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:27.215116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d539775-8971-4a45-b382-af134779e114 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:27.215611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d539775-8971-4a45-b382-af134779e114 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:27.216160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d539775-8971-4a45-b382-af134779e114 None admin] PATH_INFO: `/v3/users/e54c9872da02400cb4f4cb9e0a37bece` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:27.217493 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d539775-8971-4a45-b382-af134779e114 None admin] RBAC: Authorizing `identity:get_user(user_id=e54c9872da02400cb4f4cb9e0a37bece)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:27.219538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d539775-8971-4a45-b382-af134779e114 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:27.221651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 162/315] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:27 2022] GET /identity/v3/users/e54c9872da02400cb4f4cb9e0a37bece => generated 297 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:27.227195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7273b6d6-24c9-4798-bba4-6181dfdfdfe4 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:27.319441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:27.332427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7273b6d6-24c9-4798-bba4-6181dfdfdfe4 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:27.367294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483c7a77-0492-47f3-8ac3-b2b1c2027feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:27.367915 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483c7a77-0492-47f3-8ac3-b2b1c2027feb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:27.368195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483c7a77-0492-47f3-8ac3-b2b1c2027feb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:27.368460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483c7a77-0492-47f3-8ac3-b2b1c2027feb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:27.439327 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483c7a77-0492-47f3-8ac3-b2b1c2027feb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:27.440831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483c7a77-0492-47f3-8ac3-b2b1c2027feb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:27.529736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:27.548110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 154/316] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:27.554033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d539775-8971-4a45-b382-af134779e114 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:27.673363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d539775-8971-4a45-b382-af134779e114 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:27.707934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd54b05-7e53-4824-b3b7-43987dfec976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:27.708574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd54b05-7e53-4824-b3b7-43987dfec976 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:27.708840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd54b05-7e53-4824-b3b7-43987dfec976 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:27.709099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd54b05-7e53-4824-b3b7-43987dfec976 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:27.710230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd54b05-7e53-4824-b3b7-43987dfec976 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:27.711671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd54b05-7e53-4824-b3b7-43987dfec976 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:27.713666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 163/317] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:27 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:27.719834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483c7a77-0492-47f3-8ac3-b2b1c2027feb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:27.823224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483c7a77-0492-47f3-8ac3-b2b1c2027feb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:27.862592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd379c2-bfe7-4ed4-af91-db797206bf30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:27.863573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd379c2-bfe7-4ed4-af91-db797206bf30 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:27.864408 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd379c2-bfe7-4ed4-af91-db797206bf30 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:27.864867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd379c2-bfe7-4ed4-af91-db797206bf30 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:27.865788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd379c2-bfe7-4ed4-af91-db797206bf30 None admin] RBAC: Adding query filter params (role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3, scope.project.id=dea71e3f98bf423598a93def5ac1cd19, user.id=e54c9872da02400cb4f4cb9e0a37bece) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:27.866314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd379c2-bfe7-4ed4-af91-db797206bf30 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:27.868033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd379c2-bfe7-4ed4-af91-db797206bf30 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:27.875041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 155/318] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:27 2022] GET /identity/v3/role_assignments?user.id=e54c9872da02400cb4f4cb9e0a37bece&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:28.397178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb86580-2595-42cd-b455-a9718964e488 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:28.397729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb86580-2595-42cd-b455-a9718964e488 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:28.397729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb86580-2595-42cd-b455-a9718964e488 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:28.399085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 164/319] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:28.404656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77428eab-79df-4dd7-a9b1-fa6fea64f892 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:28.404944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77428eab-79df-4dd7-a9b1-fa6fea64f892 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:28.405215 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77428eab-79df-4dd7-a9b1-fa6fea64f892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:28.423427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-77428eab-79df-4dd7-a9b1-fa6fea64f892 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:28.424619 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-77428eab-79df-4dd7-a9b1-fa6fea64f892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:28.456437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d059966a-21a5-458f-ac1a-7692cdfb2f1e None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:28.456764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d059966a-21a5-458f-ac1a-7692cdfb2f1e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:28.456946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d059966a-21a5-458f-ac1a-7692cdfb2f1e None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:28.458319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 165/320] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:16:28.463256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b8c0b-62f9-45a0-9e8c-45fb23a87d56 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:28.463509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b8c0b-62f9-45a0-9e8c-45fb23a87d56 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:28.463807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b8c0b-62f9-45a0-9e8c-45fb23a87d56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:28.474013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:28.484353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-ce6b8c0b-62f9-45a0-9e8c-45fb23a87d56 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:28.486173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ce6b8c0b-62f9-45a0-9e8c-45fb23a87d56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:28.488293 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:28.496415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 156/321] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:28.556295 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 166/322] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:28.756719 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77428eab-79df-4dd7-a9b1-fa6fea64f892 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:28.821314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b8c0b-62f9-45a0-9e8c-45fb23a87d56 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:28.866116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77428eab-79df-4dd7-a9b1-fa6fea64f892 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:28.900374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b256827-04dd-4e91-bf64-29e4982d4bec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:28.901013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b256827-04dd-4e91-bf64-29e4982d4bec None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:28.901291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b256827-04dd-4e91-bf64-29e4982d4bec None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:28.901559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b256827-04dd-4e91-bf64-29e4982d4bec None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:28.902703 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b256827-04dd-4e91-bf64-29e4982d4bec None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:28.904156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b256827-04dd-4e91-bf64-29e4982d4bec None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:28.905749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 157/323] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:28.911589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b256827-04dd-4e91-bf64-29e4982d4bec None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:28.928471 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b8c0b-62f9-45a0-9e8c-45fb23a87d56 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:28.965670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb5afcc-1543-49c4-84d8-ecdff3b1220f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:28.966347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb5afcc-1543-49c4-84d8-ecdff3b1220f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:28.966624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb5afcc-1543-49c4-84d8-ecdff3b1220f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:28.966891 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb5afcc-1543-49c4-84d8-ecdff3b1220f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:28.975071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:28.979196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb5afcc-1543-49c4-84d8-ecdff3b1220f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-dbb5afcc-1543-49c4-84d8-ecdff3b1220f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:28.984904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:28.985965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:28.991459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 167/324] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:28 2022] GET /identity/v3/roles/reader => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:16:28.993389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb5afcc-1543-49c4-84d8-ecdff3b1220f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:29.026364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b256827-04dd-4e91-bf64-29e4982d4bec None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:29.061949 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb59a14a-6c8c-4246-8ca4-89b06afdb4c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:29.062662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb59a14a-6c8c-4246-8ca4-89b06afdb4c5 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:29.062945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb59a14a-6c8c-4246-8ca4-89b06afdb4c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:29.063214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb59a14a-6c8c-4246-8ca4-89b06afdb4c5 None admin] PATH_INFO: `/v3/groups` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:29.063854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb59a14a-6c8c-4246-8ca4-89b06afdb4c5 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:29.065309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb59a14a-6c8c-4246-8ca4-89b06afdb4c5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:29.081656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 158/325] 10.222.0.16 () {60 vars in 1134 bytes} [Tue Sep 13 15:16:28 2022] POST /identity/v3/groups => generated 226 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:16:29.111469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb5afcc-1543-49c4-84d8-ecdff3b1220f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:29.121602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e855b83-270e-418c-956d-c365386b1f14 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:29.121908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e855b83-270e-418c-956d-c365386b1f14 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:29.122166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e855b83-270e-418c-956d-c365386b1f14 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:29.123727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 159/326] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:16:29.129550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a77a50-c0ea-4ab1-b02b-c943473f1cd2 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:29.129885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a77a50-c0ea-4ab1-b02b-c943473f1cd2 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:29.130112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a77a50-c0ea-4ab1-b02b-c943473f1cd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:29.149379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-b6a77a50-c0ea-4ab1-b02b-c943473f1cd2 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:29.150823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b6a77a50-c0ea-4ab1-b02b-c943473f1cd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:29.155173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe6c97d-f999-473d-9c01-b313aed9d193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:29.156102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe6c97d-f999-473d-9c01-b313aed9d193 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:29.156566 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe6c97d-f999-473d-9c01-b313aed9d193 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:29.156913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe6c97d-f999-473d-9c01-b313aed9d193 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:29.157669 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe6c97d-f999-473d-9c01-b313aed9d193 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:29.158046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe6c97d-f999-473d-9c01-b313aed9d193 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:29.159596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe6c97d-f999-473d-9c01-b313aed9d193 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:29.169901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 168/327] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:28 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:29.175905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe6c97d-f999-473d-9c01-b313aed9d193 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:29.234607 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 160/328] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:29.253039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:29.300850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe6c97d-f999-473d-9c01-b313aed9d193 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:29.337783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbf3890-f309-488e-b72b-628b3b0df65d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:29.338511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbf3890-f309-488e-b72b-628b3b0df65d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:29.338791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbf3890-f309-488e-b72b-628b3b0df65d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:29.339105 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbf3890-f309-488e-b72b-628b3b0df65d None admin] PATH_INFO: `/v3/users/f223de0989bd4b978ec0f8ec8e64f728` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:29.340230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbf3890-f309-488e-b72b-628b3b0df65d None admin] RBAC: Authorizing `identity:get_user(user_id=f223de0989bd4b978ec0f8ec8e64f728)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:29.341660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbf3890-f309-488e-b72b-628b3b0df65d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:29.343341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 169/329] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:29 2022] GET /identity/v3/users/f223de0989bd4b978ec0f8ec8e64f728 => generated 293 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:29.349372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a77a50-c0ea-4ab1-b02b-c943473f1cd2 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:29.395826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:29.462564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a77a50-c0ea-4ab1-b02b-c943473f1cd2 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:29.499212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f46c839-693c-4a96-a619-898c27d048fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:29.499989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f46c839-693c-4a96-a619-898c27d048fc None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:29.500244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f46c839-693c-4a96-a619-898c27d048fc None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:29.500520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f46c839-693c-4a96-a619-898c27d048fc None admin] PATH_INFO: `/v3/system/users/f223de0989bd4b978ec0f8ec8e64f728/roles/0ff8d8f3f5ae426197cdffd9ea41a4e3` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:29.502036 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f46c839-693c-4a96-a619-898c27d048fc None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=f223de0989bd4b978ec0f8ec8e64f728, role_id=0ff8d8f3f5ae426197cdffd9ea41a4e3)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:29.503473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f46c839-693c-4a96-a619-898c27d048fc None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:29.510265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbf3890-f309-488e-b72b-628b3b0df65d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:29.513286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 161/330] 10.222.0.16 () {58 vars in 1334 bytes} [Tue Sep 13 15:16:29 2022] PUT /identity/v3/system/users/f223de0989bd4b978ec0f8ec8e64f728/roles/0ff8d8f3f5ae426197cdffd9ea41a4e3 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:29.643108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbf3890-f309-488e-b72b-628b3b0df65d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:29.675076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:29.683975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e6ba71-1b1b-4365-8e1a-e01da87acc65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:29.684631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6ba71-1b1b-4365-8e1a-e01da87acc65 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:29.684905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6ba71-1b1b-4365-8e1a-e01da87acc65 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:29.685171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6ba71-1b1b-4365-8e1a-e01da87acc65 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:29.698517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e6ba71-1b1b-4365-8e1a-e01da87acc65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-91e6ba71-1b1b-4365-8e1a-e01da87acc65 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:29.704211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:29.705365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:29.706316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 170/331] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:16:29.711428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f46c839-693c-4a96-a619-898c27d048fc None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:29.820649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f46c839-693c-4a96-a619-898c27d048fc None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:29.823097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:29.860017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd0f078-9165-462a-b5e6-4ae0ac7bebf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:29.860642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0f078-9165-462a-b5e6-4ae0ac7bebf9 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:29.860922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0f078-9165-462a-b5e6-4ae0ac7bebf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:29.861194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0f078-9165-462a-b5e6-4ae0ac7bebf9 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:29.861864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd0f078-9165-462a-b5e6-4ae0ac7bebf9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:29.862192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd0f078-9165-462a-b5e6-4ae0ac7bebf9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:29.863614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd0f078-9165-462a-b5e6-4ae0ac7bebf9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:29.873160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 162/332] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:29.879108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e6ba71-1b1b-4365-8e1a-e01da87acc65 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:29.983842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e6ba71-1b1b-4365-8e1a-e01da87acc65 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:30.021659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cd3625-f71a-4ff7-9fab-900f3ebb713f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:30.022346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd3625-f71a-4ff7-9fab-900f3ebb713f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:30.022629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd3625-f71a-4ff7-9fab-900f3ebb713f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:30.022892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cd3625-f71a-4ff7-9fab-900f3ebb713f None admin] PATH_INFO: `/v3/users/9dfc400edaaf43cb9c7cb54740656bc8` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:30.024057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd3625-f71a-4ff7-9fab-900f3ebb713f None admin] RBAC: Authorizing `identity:get_user(user_id=9dfc400edaaf43cb9c7cb54740656bc8)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:30.025639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cd3625-f71a-4ff7-9fab-900f3ebb713f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:30.027654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 171/333] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:29 2022] GET /identity/v3/users/9dfc400edaaf43cb9c7cb54740656bc8 => generated 293 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:30.033681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd0f078-9165-462a-b5e6-4ae0ac7bebf9 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:30.153560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd0f078-9165-462a-b5e6-4ae0ac7bebf9 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:30.203148 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2504b046-2bd2-4a04-ab17-88c06b18b9c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:30.203776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2504b046-2bd2-4a04-ab17-88c06b18b9c4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:30.204182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2504b046-2bd2-4a04-ab17-88c06b18b9c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:30.204459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2504b046-2bd2-4a04-ab17-88c06b18b9c4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:30.205195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2504b046-2bd2-4a04-ab17-88c06b18b9c4 None admin] RBAC: Adding query filter params (role.id=7a7e816668a7493bad0752baca37934c, user.id=9dfc400edaaf43cb9c7cb54740656bc8, scope.system=all) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:30.205501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2504b046-2bd2-4a04-ab17-88c06b18b9c4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:30.206986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2504b046-2bd2-4a04-ab17-88c06b18b9c4 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:30.226959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 163/334] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Sep 13 15:16:30 2022] GET /identity/v3/role_assignments?user.id=9dfc400edaaf43cb9c7cb54740656bc8&scope.system=all&role.id=7a7e816668a7493bad0752baca37934c => generated 510 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:31.972618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a87f3-4e54-435d-8ee5-aa6b74bf8bd1 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:31.973520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a87f3-4e54-435d-8ee5-aa6b74bf8bd1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:31.973938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2a87f3-4e54-435d-8ee5-aa6b74bf8bd1 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:31.975702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 172/335] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:31.981470 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e03c881-eeeb-483c-8963-a7e7722d3eaf None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:31.982249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e03c881-eeeb-483c-8963-a7e7722d3eaf None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:31.982668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e03c881-eeeb-483c-8963-a7e7722d3eaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:32.017177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-4e03c881-eeeb-483c-8963-a7e7722d3eaf None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:32.018649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4e03c881-eeeb-483c-8963-a7e7722d3eaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:32.043736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:32.069686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b999d159-5e17-40f7-8fb6-cacbb025ea27 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:32.069970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b999d159-5e17-40f7-8fb6-cacbb025ea27 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:32.070273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b999d159-5e17-40f7-8fb6-cacbb025ea27 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:32.071925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 173/336] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:32.080272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e281ab-2235-45fe-b533-85d278bf4898 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:32.080559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e281ab-2235-45fe-b533-85d278bf4898 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:32.080829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e281ab-2235-45fe-b533-85d278bf4898 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:32.111264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d2e281ab-2235-45fe-b533-85d278bf4898 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:32.112509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d2e281ab-2235-45fe-b533-85d278bf4898 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:32.116828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 164/337] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:32.184081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:32.202174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 174/338] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:32.687299 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e03c881-eeeb-483c-8963-a7e7722d3eaf None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:32.744888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fda248-2d9c-4f8e-9564-b9eeeabe60b2 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:32.745194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fda248-2d9c-4f8e-9564-b9eeeabe60b2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:32.745451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fda248-2d9c-4f8e-9564-b9eeeabe60b2 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:32.746932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 175/339] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:32.747543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fda248-2d9c-4f8e-9564-b9eeeabe60b2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:32.834047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e03c881-eeeb-483c-8963-a7e7722d3eaf None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:32.897308 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a71c37b-21d6-492a-8c2a-24ef4f351c63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:32.897968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a71c37b-21d6-492a-8c2a-24ef4f351c63 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:32.898267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a71c37b-21d6-492a-8c2a-24ef4f351c63 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:32.898531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a71c37b-21d6-492a-8c2a-24ef4f351c63 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:32.917853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a71c37b-21d6-492a-8c2a-24ef4f351c63 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-9a71c37b-21d6-492a-8c2a-24ef4f351c63 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:32.923301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:32.924184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:32.924962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:32.929071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 165/340] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:32.929804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fda248-2d9c-4f8e-9564-b9eeeabe60b2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:32.930473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ef9132-4366-4c46-8c7b-c8ec94b2dddb None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:32.931312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ef9132-4366-4c46-8c7b-c8ec94b2dddb None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:32.931590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ef9132-4366-4c46-8c7b-c8ec94b2dddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:32.959875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e9ef9132-4366-4c46-8c7b-c8ec94b2dddb None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:32.960622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e9ef9132-4366-4c46-8c7b-c8ec94b2dddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:32.972018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b558576f-07d6-4dd6-b078-d1ac7637708b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:32.972992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558576f-07d6-4dd6-b078-d1ac7637708b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:32.974420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558576f-07d6-4dd6-b078-d1ac7637708b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:32.974902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558576f-07d6-4dd6-b078-d1ac7637708b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:32.978138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:32.996019 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b558576f-07d6-4dd6-b078-d1ac7637708b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-b558576f-07d6-4dd6-b078-d1ac7637708b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:33.003398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:33.004294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 13 15:16:33.005060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:33.009055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 176/341] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:32 2022] GET /identity/v3/roles/reader => generated 84 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:33.011078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b558576f-07d6-4dd6-b078-d1ac7637708b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:33.059749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 166/342] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:33.061324 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ef9132-4366-4c46-8c7b-c8ec94b2dddb None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:33.082455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:33.179669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ef9132-4366-4c46-8c7b-c8ec94b2dddb None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:33.188497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b558576f-07d6-4dd6-b078-d1ac7637708b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:33.220447 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cfbfe8-d96b-49b0-b4ef-a120e900e4df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:33.221552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cfbfe8-d96b-49b0-b4ef-a120e900e4df None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:33.221963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cfbfe8-d96b-49b0-b4ef-a120e900e4df None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:33.222432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cfbfe8-d96b-49b0-b4ef-a120e900e4df None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:33.223229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cfbfe8-d96b-49b0-b4ef-a120e900e4df None admin] RBAC: Adding query filter params (name=reader) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:33.223685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cfbfe8-d96b-49b0-b4ef-a120e900e4df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:33.225259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cfbfe8-d96b-49b0-b4ef-a120e900e4df None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:33.234802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 167/343] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:33.236546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d3a9ef-6f37-4573-a050-296e445af258 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:33.236963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d3a9ef-6f37-4573-a050-296e445af258 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:33.237389 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d3a9ef-6f37-4573-a050-296e445af258 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:33.239772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 168/344] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:33.246224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9df2af-832c-4116-b632-81f8ca04ddf2 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:33.246665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9df2af-832c-4116-b632-81f8ca04ddf2 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:33.247078 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9df2af-832c-4116-b632-81f8ca04ddf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:33.264888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:33.270868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce0c71b-fcc4-4273-b6f1-3f265a7edece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:33.271511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce0c71b-fcc4-4273-b6f1-3f265a7edece None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:33.271790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce0c71b-fcc4-4273-b6f1-3f265a7edece None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:33.272050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce0c71b-fcc4-4273-b6f1-3f265a7edece None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:33.272412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8b9df2af-832c-4116-b632-81f8ca04ddf2 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:33.272735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce0c71b-fcc4-4273-b6f1-3f265a7edece None admin] RBAC: Adding query filter params (name=member) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:33.273044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce0c71b-fcc4-4273-b6f1-3f265a7edece None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:33.274263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8b9df2af-832c-4116-b632-81f8ca04ddf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:33.274635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce0c71b-fcc4-4273-b6f1-3f265a7edece None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:33.284176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 177/345] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:33.286364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce0c71b-fcc4-4273-b6f1-3f265a7edece None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:33.399923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 169/346] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:33.400875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9df2af-832c-4116-b632-81f8ca04ddf2 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:33.448238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce0c71b-fcc4-4273-b6f1-3f265a7edece None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:33.461243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:33.491199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f191c23-38c2-4ded-9ee8-78f3634c0daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:33.491850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f191c23-38c2-4ded-9ee8-78f3634c0daa None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:33.492126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f191c23-38c2-4ded-9ee8-78f3634c0daa None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:33.492397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f191c23-38c2-4ded-9ee8-78f3634c0daa None admin] PATH_INFO: `/v3/users/f223de0989bd4b978ec0f8ec8e64f728` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:33.493547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f191c23-38c2-4ded-9ee8-78f3634c0daa None admin] RBAC: Authorizing `identity:get_user(user_id=f223de0989bd4b978ec0f8ec8e64f728)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:33.495608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f191c23-38c2-4ded-9ee8-78f3634c0daa None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:33.498359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 178/347] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:16:33 2022] GET /identity/v3/users/f223de0989bd4b978ec0f8ec8e64f728 => generated 293 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:33.499266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f191c23-38c2-4ded-9ee8-78f3634c0daa None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:33.522233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9df2af-832c-4116-b632-81f8ca04ddf2 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:33.570184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3755b44-b5b4-4e21-aa87-6f48b88b2297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:33.571163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3755b44-b5b4-4e21-aa87-6f48b88b2297 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:33.571688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3755b44-b5b4-4e21-aa87-6f48b88b2297 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:33.572242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3755b44-b5b4-4e21-aa87-6f48b88b2297 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:33.580331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:33.650131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f191c23-38c2-4ded-9ee8-78f3634c0daa None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:33.701478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3755b44-b5b4-4e21-aa87-6f48b88b2297 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:33.703406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3755b44-b5b4-4e21-aa87-6f48b88b2297 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:33.721583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5389279b-600d-4092-aa58-257edad41914 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:33.722274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5389279b-600d-4092-aa58-257edad41914 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:33.722552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5389279b-600d-4092-aa58-257edad41914 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:33.722822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5389279b-600d-4092-aa58-257edad41914 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:33.739665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5389279b-600d-4092-aa58-257edad41914 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-5389279b-600d-4092-aa58-257edad41914 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:33.744932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:33.745899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:33.746758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 179/348] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:33.751940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5389279b-600d-4092-aa58-257edad41914 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:33.801769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:33.822141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 170/349] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:33.824990 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b0746-ff61-4a3f-97c8-b62510bed866 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:33.825583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b0746-ff61-4a3f-97c8-b62510bed866 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:33.826051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b0746-ff61-4a3f-97c8-b62510bed866 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:33.828229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 171/350] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:33.829331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740b0746-ff61-4a3f-97c8-b62510bed866 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:33.876000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5389279b-600d-4092-aa58-257edad41914 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:33.878413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:33.925302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3edd144-af64-4b68-a4b3-289dfd1fac6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:33.925956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3edd144-af64-4b68-a4b3-289dfd1fac6a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:33.926267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3edd144-af64-4b68-a4b3-289dfd1fac6a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:33.926543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3edd144-af64-4b68-a4b3-289dfd1fac6a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:33.927283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3edd144-af64-4b68-a4b3-289dfd1fac6a None admin] RBAC: Adding query filter params (role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3, user.id=f223de0989bd4b978ec0f8ec8e64f728, scope.system=all) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:33.927594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3edd144-af64-4b68-a4b3-289dfd1fac6a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:33.929507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3edd144-af64-4b68-a4b3-289dfd1fac6a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:33.943942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 180/351] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Sep 13 15:16:33 2022] GET /identity/v3/role_assignments?user.id=f223de0989bd4b978ec0f8ec8e64f728&scope.system=all&role.id=0ff8d8f3f5ae426197cdffd9ea41a4e3 => generated 510 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:33.945485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3edd144-af64-4b68-a4b3-289dfd1fac6a None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:33.964571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740b0746-ff61-4a3f-97c8-b62510bed866 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:34.016933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af7b8f-528a-4e01-9fba-cc024a0eade5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:34.018780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af7b8f-528a-4e01-9fba-cc024a0eade5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.019535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af7b8f-528a-4e01-9fba-cc024a0eade5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.020065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af7b8f-528a-4e01-9fba-cc024a0eade5 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.021135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24af7b8f-528a-4e01-9fba-cc024a0eade5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:34.021716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24af7b8f-528a-4e01-9fba-cc024a0eade5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:34.023722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24af7b8f-528a-4e01-9fba-cc024a0eade5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:34.036909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 172/352] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:34.043250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af7b8f-528a-4e01-9fba-cc024a0eade5 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:34.083541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:34.092221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3edd144-af64-4b68-a4b3-289dfd1fac6a None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:34.141185 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5bda18-2c59-4a7a-a59c-9c720a95ce62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:34.142406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5bda18-2c59-4a7a-a59c-9c720a95ce62 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.142696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5bda18-2c59-4a7a-a59c-9c720a95ce62 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.142968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5bda18-2c59-4a7a-a59c-9c720a95ce62 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.160559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5bda18-2c59-4a7a-a59c-9c720a95ce62 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-8d5bda18-2c59-4a7a-a59c-9c720a95ce62 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:34.167177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:34.168100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:34.168865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 181/353] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:34.170017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e2c82-e923-4321-b11f-485e34a9a5db None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.170017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e2c82-e923-4321-b11f-485e34a9a5db None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.171774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e2c82-e923-4321-b11f-485e34a9a5db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.192306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7d1e2c82-e923-4321-b11f-485e34a9a5db None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:34.193696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7d1e2c82-e923-4321-b11f-485e34a9a5db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:34.200847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:34.207523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af7b8f-528a-4e01-9fba-cc024a0eade5 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:34.253478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1591a771-09b5-49e2-af80-80e52e5642fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:34.254086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1591a771-09b5-49e2-af80-80e52e5642fb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.254415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1591a771-09b5-49e2-af80-80e52e5642fb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.254686 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1591a771-09b5-49e2-af80-80e52e5642fb None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.256676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1591a771-09b5-49e2-af80-80e52e5642fb None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:34.258226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1591a771-09b5-49e2-af80-80e52e5642fb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:34.259917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 173/354] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:34 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:34.260680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1591a771-09b5-49e2-af80-80e52e5642fb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:34.279153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 182/355] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:34.285128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1e2c82-e923-4321-b11f-485e34a9a5db None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:34.389266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1591a771-09b5-49e2-af80-80e52e5642fb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:34.392829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:34.412882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1e2c82-e923-4321-b11f-485e34a9a5db None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:34.426464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cd686a-52ec-45f9-9280-bdfb78f9acd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:34.428337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cd686a-52ec-45f9-9280-bdfb78f9acd1 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.428806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cd686a-52ec-45f9-9280-bdfb78f9acd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.429274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cd686a-52ec-45f9-9280-bdfb78f9acd1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.449729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b31f90-9ebf-4e35-ad98-761126763332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:34.450377 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b31f90-9ebf-4e35-ad98-761126763332 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.450658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b31f90-9ebf-4e35-ad98-761126763332 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.450979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b31f90-9ebf-4e35-ad98-761126763332 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.451646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b31f90-9ebf-4e35-ad98-761126763332 None admin] RBAC: Adding query filter params (name=member) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:34.451990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b31f90-9ebf-4e35-ad98-761126763332 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:34.453544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b31f90-9ebf-4e35-ad98-761126763332 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:34.463851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 183/356] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:34.465125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c3459c-3179-4858-af55-4cfc05b8ac06 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.465509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c3459c-3179-4858-af55-4cfc05b8ac06 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.465723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c3459c-3179-4858-af55-4cfc05b8ac06 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.466986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 184/357] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:34.467787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c3459c-3179-4858-af55-4cfc05b8ac06 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:34.468106 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:34.513130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cd686a-52ec-45f9-9280-bdfb78f9acd1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:34.514795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cd686a-52ec-45f9-9280-bdfb78f9acd1 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:34.606913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c3459c-3179-4858-af55-4cfc05b8ac06 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:34.636511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 174/358] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:34.638017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a671777-842e-4a52-818f-5b3e0ba80e4a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.638322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a671777-842e-4a52-818f-5b3e0ba80e4a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.638589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a671777-842e-4a52-818f-5b3e0ba80e4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.651282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3368a7f7-dc3f-48f6-a7fa-bea22a5d3659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:34.651927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3368a7f7-dc3f-48f6-a7fa-bea22a5d3659 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.652202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3368a7f7-dc3f-48f6-a7fa-bea22a5d3659 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.652471 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3368a7f7-dc3f-48f6-a7fa-bea22a5d3659 None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.659356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8a671777-842e-4a52-818f-5b3e0ba80e4a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:34.659745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3368a7f7-dc3f-48f6-a7fa-bea22a5d3659 None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:34.660559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8a671777-842e-4a52-818f-5b3e0ba80e4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:34.661195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3368a7f7-dc3f-48f6-a7fa-bea22a5d3659 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:34.663201 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 185/359] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:34 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:34.664034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3368a7f7-dc3f-48f6-a7fa-bea22a5d3659 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:34.678919 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:34.734904 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:34.744443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 175/360] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:34.745298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a671777-842e-4a52-818f-5b3e0ba80e4a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:34.790758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3368a7f7-dc3f-48f6-a7fa-bea22a5d3659 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:34.829665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178126b1-ede4-4904-ad81-b555d215b033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:34.830327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178126b1-ede4-4904-ad81-b555d215b033 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.830608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178126b1-ede4-4904-ad81-b555d215b033 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.830881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178126b1-ede4-4904-ad81-b555d215b033 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.866978 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a671777-842e-4a52-818f-5b3e0ba80e4a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:34.915120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0858c742-a067-4d66-b1c2-e544056eb283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:34.915120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0858c742-a067-4d66-b1c2-e544056eb283 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:34.915497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0858c742-a067-4d66-b1c2-e544056eb283 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:34.915497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0858c742-a067-4d66-b1c2-e544056eb283 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:34.921131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178126b1-ede4-4904-ad81-b555d215b033 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:34.923034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178126b1-ede4-4904-ad81-b555d215b033 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:34.928838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0858c742-a067-4d66-b1c2-e544056eb283 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-0858c742-a067-4d66-b1c2-e544056eb283 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:34.935042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:34.935946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:34.936810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:34.937469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:34.939368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 176/361] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:34.940366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0858c742-a067-4d66-b1c2-e544056eb283 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:35.009487 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:35.056199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 186/362] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:35.057880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178126b1-ede4-4904-ad81-b555d215b033 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:35.059418 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0858c742-a067-4d66-b1c2-e544056eb283 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:35.100112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b90ddf-51bb-4998-a9e0-33916c4f25f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:35.100758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b90ddf-51bb-4998-a9e0-33916c4f25f8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:35.101028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b90ddf-51bb-4998-a9e0-33916c4f25f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:35.101293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b90ddf-51bb-4998-a9e0-33916c4f25f8 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:35.102683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b90ddf-51bb-4998-a9e0-33916c4f25f8 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:35.104165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b90ddf-51bb-4998-a9e0-33916c4f25f8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:35.105907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 177/363] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:34 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:35.106631 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b90ddf-51bb-4998-a9e0-33916c4f25f8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:35.185918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178126b1-ede4-4904-ad81-b555d215b033 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:35.198314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:35.224993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b90ddf-51bb-4998-a9e0-33916c4f25f8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:35.227408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0303f3-6ec4-4e5c-8704-1e188cdc2b0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:35.228031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0303f3-6ec4-4e5c-8704-1e188cdc2b0b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:35.228303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0303f3-6ec4-4e5c-8704-1e188cdc2b0b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:35.228566 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0303f3-6ec4-4e5c-8704-1e188cdc2b0b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:35.229285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0303f3-6ec4-4e5c-8704-1e188cdc2b0b None admin] RBAC: Adding query filter params (group.id=69bbd4b43fc64649997434589044be4c, role.id=7a7e816668a7493bad0752baca37934c, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:35.229590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0303f3-6ec4-4e5c-8704-1e188cdc2b0b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:35.231058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0303f3-6ec4-4e5c-8704-1e188cdc2b0b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:35.238397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 187/364] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Sep 13 15:16:35 2022] GET /identity/v3/role_assignments?group.id=69bbd4b43fc64649997434589044be4c&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=7a7e816668a7493bad0752baca37934c => generated 260 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:35.239651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0303f3-6ec4-4e5c-8704-1e188cdc2b0b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:35.259086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:35.277914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db102b50-d267-4cb5-b55d-e86ec3c95d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:35.279226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db102b50-d267-4cb5-b55d-e86ec3c95d20 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:35.280315 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db102b50-d267-4cb5-b55d-e86ec3c95d20 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:35.281095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db102b50-d267-4cb5-b55d-e86ec3c95d20 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:35.282897 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db102b50-d267-4cb5-b55d-e86ec3c95d20 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:35.283216 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db102b50-d267-4cb5-b55d-e86ec3c95d20 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:35.284678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db102b50-d267-4cb5-b55d-e86ec3c95d20 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:35.294939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 178/365] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:35.297601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db102b50-d267-4cb5-b55d-e86ec3c95d20 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:35.372828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0303f3-6ec4-4e5c-8704-1e188cdc2b0b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:35.423948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252ed9c9-6958-4951-9fa1-70b6d76986e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:35.425519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252ed9c9-6958-4951-9fa1-70b6d76986e6 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:35.425981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252ed9c9-6958-4951-9fa1-70b6d76986e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:35.426638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252ed9c9-6958-4951-9fa1-70b6d76986e6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:35.429889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db102b50-d267-4cb5-b55d-e86ec3c95d20 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:35.441642 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252ed9c9-6958-4951-9fa1-70b6d76986e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-252ed9c9-6958-4951-9fa1-70b6d76986e6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:35.448174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:35.449261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:35.450174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:35.452132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 188/366] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:16:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:35.453233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252ed9c9-6958-4951-9fa1-70b6d76986e6 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:35.473490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162ba052-4f9f-468e-9185-fc7b438bf0bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:35.475075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162ba052-4f9f-468e-9185-fc7b438bf0bc None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:35.475075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162ba052-4f9f-468e-9185-fc7b438bf0bc None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:35.475075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162ba052-4f9f-468e-9185-fc7b438bf0bc None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:35.475776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162ba052-4f9f-468e-9185-fc7b438bf0bc None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:35.477227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162ba052-4f9f-468e-9185-fc7b438bf0bc None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:35.479989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 179/367] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:35 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:35.479989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162ba052-4f9f-468e-9185-fc7b438bf0bc None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:35.499491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:35.557861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:35.588693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252ed9c9-6958-4951-9fa1-70b6d76986e6 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:35.622641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162ba052-4f9f-468e-9185-fc7b438bf0bc None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:35.634164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424baaa2-3557-48a3-93e9-52e4d6318dcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:35.634753 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424baaa2-3557-48a3-93e9-52e4d6318dcd None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:35.635030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424baaa2-3557-48a3-93e9-52e4d6318dcd None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:35.635306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424baaa2-3557-48a3-93e9-52e4d6318dcd None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:35.637093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424baaa2-3557-48a3-93e9-52e4d6318dcd None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:35.638665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424baaa2-3557-48a3-93e9-52e4d6318dcd None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:35.641307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 189/368] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:35 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:35.641706 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424baaa2-3557-48a3-93e9-52e4d6318dcd None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:35.668459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442e1754-4b2d-4ced-b556-1960f13f9513 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:35.669182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e1754-4b2d-4ced-b556-1960f13f9513 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:35.669406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e1754-4b2d-4ced-b556-1960f13f9513 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:35.669686 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442e1754-4b2d-4ced-b556-1960f13f9513 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:35.749338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442e1754-4b2d-4ced-b556-1960f13f9513 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:35.750986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442e1754-4b2d-4ced-b556-1960f13f9513 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:35.785080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424baaa2-3557-48a3-93e9-52e4d6318dcd None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:35.786938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:35.825047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:35.828297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04d8296-e559-4ad9-b1cd-3676b86a2b84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:35.828297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04d8296-e559-4ad9-b1cd-3676b86a2b84 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:35.828297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04d8296-e559-4ad9-b1cd-3676b86a2b84 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:35.828297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04d8296-e559-4ad9-b1cd-3676b86a2b84 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:35.828685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04d8296-e559-4ad9-b1cd-3676b86a2b84 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:35.829053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04d8296-e559-4ad9-b1cd-3676b86a2b84 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:35.830452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04d8296-e559-4ad9-b1cd-3676b86a2b84 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:35.840851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 190/369] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:35.841518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04d8296-e559-4ad9-b1cd-3676b86a2b84 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:35.881329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 180/370] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:35.882621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442e1754-4b2d-4ced-b556-1960f13f9513 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:35.978773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04d8296-e559-4ad9-b1cd-3676b86a2b84 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:36.007390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442e1754-4b2d-4ced-b556-1960f13f9513 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:36.016268 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebeeb949-f092-43c5-a7bf-7a696716f155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:36.017241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeeb949-f092-43c5-a7bf-7a696716f155 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:36.017532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeeb949-f092-43c5-a7bf-7a696716f155 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:36.017804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeeb949-f092-43c5-a7bf-7a696716f155 None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:36.019275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebeeb949-f092-43c5-a7bf-7a696716f155 None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:36.020632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebeeb949-f092-43c5-a7bf-7a696716f155 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:36.022223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 191/371] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:35 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:36.027005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebeeb949-f092-43c5-a7bf-7a696716f155 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:36.042867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aaf151-3d7e-4cb1-9593-69663f6195ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:36.043594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aaf151-3d7e-4cb1-9593-69663f6195ea None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:36.043873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aaf151-3d7e-4cb1-9593-69663f6195ea None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:36.044264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aaf151-3d7e-4cb1-9593-69663f6195ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:36.044993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96aaf151-3d7e-4cb1-9593-69663f6195ea None admin] RBAC: Adding query filter params (group.id=69bbd4b43fc64649997434589044be4c, role.id=74db8755b1f04d839260ccd3e36deb66, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:36.045304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96aaf151-3d7e-4cb1-9593-69663f6195ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:36.046860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96aaf151-3d7e-4cb1-9593-69663f6195ea None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:36.062207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 181/372] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Sep 13 15:16:35 2022] GET /identity/v3/role_assignments?group.id=69bbd4b43fc64649997434589044be4c&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=74db8755b1f04d839260ccd3e36deb66 => generated 260 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:36.062207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aaf151-3d7e-4cb1-9593-69663f6195ea None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:36.071162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:36.094873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:36.165963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebeeb949-f092-43c5-a7bf-7a696716f155 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:36.212984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae204ce-33b1-4dda-81c4-984ffdb312ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:36.213716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae204ce-33b1-4dda-81c4-984ffdb312ba None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:36.214167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae204ce-33b1-4dda-81c4-984ffdb312ba None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:36.214512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae204ce-33b1-4dda-81c4-984ffdb312ba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:36.216611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aaf151-3d7e-4cb1-9593-69663f6195ea None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:36.263019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7523d99-3817-49ad-827e-e0532877712f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:36.263635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7523d99-3817-49ad-827e-e0532877712f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:36.263854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7523d99-3817-49ad-827e-e0532877712f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:36.264133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7523d99-3817-49ad-827e-e0532877712f None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:36.265528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7523d99-3817-49ad-827e-e0532877712f None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:36.267606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7523d99-3817-49ad-827e-e0532877712f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:36.270423 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 182/373] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:36 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:36.271487 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7523d99-3817-49ad-827e-e0532877712f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:36.301123 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae204ce-33b1-4dda-81c4-984ffdb312ba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:36.303571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae204ce-33b1-4dda-81c4-984ffdb312ba None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:36.341746 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:36.376247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:36.396649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7523d99-3817-49ad-827e-e0532877712f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:36.433151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 192/374] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:36.434030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae204ce-33b1-4dda-81c4-984ffdb312ba None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:36.446425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb9a442-034c-4051-b056-f9dcaf9da99a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:36.447405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb9a442-034c-4051-b056-f9dcaf9da99a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:36.447685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb9a442-034c-4051-b056-f9dcaf9da99a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:36.447964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb9a442-034c-4051-b056-f9dcaf9da99a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:36.449298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb9a442-034c-4051-b056-f9dcaf9da99a None admin] RBAC: Adding query filter params (group.id=69bbd4b43fc64649997434589044be4c, role.id=7a7e816668a7493bad0752baca37934c, scope.project.id=dea71e3f98bf423598a93def5ac1cd19) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:36.449625 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb9a442-034c-4051-b056-f9dcaf9da99a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:36.451292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb9a442-034c-4051-b056-f9dcaf9da99a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:36.458379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 183/375] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Sep 13 15:16:36 2022] GET /identity/v3/role_assignments?group.id=69bbd4b43fc64649997434589044be4c&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=7a7e816668a7493bad0752baca37934c => generated 260 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:36.458993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb9a442-034c-4051-b056-f9dcaf9da99a None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:36.580301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae204ce-33b1-4dda-81c4-984ffdb312ba None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:36.618234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ab827b-aa25-46fe-8ce4-104219d2a235 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:36.618928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ab827b-aa25-46fe-8ce4-104219d2a235 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:36.619213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ab827b-aa25-46fe-8ce4-104219d2a235 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:36.619494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ab827b-aa25-46fe-8ce4-104219d2a235 None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:36.621044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ab827b-aa25-46fe-8ce4-104219d2a235 None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:36.622599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ab827b-aa25-46fe-8ce4-104219d2a235 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:36.624445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 193/376] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:36 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:36.624936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:36.633200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ab827b-aa25-46fe-8ce4-104219d2a235 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:36.688124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb9a442-034c-4051-b056-f9dcaf9da99a None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:36.725537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e22caf-6678-4f6e-b0e0-461a8995226d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:36.726499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e22caf-6678-4f6e-b0e0-461a8995226d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:36.727363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e22caf-6678-4f6e-b0e0-461a8995226d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:36.727742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e22caf-6678-4f6e-b0e0-461a8995226d None admin] PATH_INFO: `/v3/projects/3dcdfc529b6149688ceee6b949a735a1` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:36.729108 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e22caf-6678-4f6e-b0e0-461a8995226d None admin] RBAC: Authorizing `identity:get_project(project_id=3dcdfc529b6149688ceee6b949a735a1)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:36.730688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e22caf-6678-4f6e-b0e0-461a8995226d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:36.732894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 184/377] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:36 2022] GET /identity/v3/projects/3dcdfc529b6149688ceee6b949a735a1 => generated 344 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:36.739420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e22caf-6678-4f6e-b0e0-461a8995226d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:36.775980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:36.794796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ab827b-aa25-46fe-8ce4-104219d2a235 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:36.860968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a7026-58be-4e56-92cf-ee2ab4690dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:36.861507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a7026-58be-4e56-92cf-ee2ab4690dfa None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:36.861782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a7026-58be-4e56-92cf-ee2ab4690dfa None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:36.862049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73a7026-58be-4e56-92cf-ee2ab4690dfa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:36.862819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a7026-58be-4e56-92cf-ee2ab4690dfa None admin] RBAC: Adding query filter params (group.id=69bbd4b43fc64649997434589044be4c, role.id=74db8755b1f04d839260ccd3e36deb66, scope.project.id=dea71e3f98bf423598a93def5ac1cd19) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:36.863128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a7026-58be-4e56-92cf-ee2ab4690dfa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:36.865138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73a7026-58be-4e56-92cf-ee2ab4690dfa None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:36.880723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 194/378] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Sep 13 15:16:36 2022] GET /identity/v3/role_assignments?group.id=69bbd4b43fc64649997434589044be4c&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=74db8755b1f04d839260ccd3e36deb66 => generated 260 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:36.921682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e22caf-6678-4f6e-b0e0-461a8995226d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:36.976680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069b1893-1055-4a15-9cd8-df0ff2ae11a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:36.977363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b1893-1055-4a15-9cd8-df0ff2ae11a0 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:36.977643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b1893-1055-4a15-9cd8-df0ff2ae11a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:36.977917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069b1893-1055-4a15-9cd8-df0ff2ae11a0 None admin] PATH_INFO: `/v3/groups/f3b2710f65d74a5a8180706d20789222` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:36.989261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069b1893-1055-4a15-9cd8-df0ff2ae11a0 None admin] RBAC: Authorizing `identity:get_group(group_id=f3b2710f65d74a5a8180706d20789222)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:36.990787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069b1893-1055-4a15-9cd8-df0ff2ae11a0 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:36.992926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 185/379] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:36 2022] GET /identity/v3/groups/f3b2710f65d74a5a8180706d20789222 => generated 229 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:36.999224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a7026-58be-4e56-92cf-ee2ab4690dfa None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:37.076646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:37.124434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73a7026-58be-4e56-92cf-ee2ab4690dfa None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:37.165467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5515c5-7415-4c09-9704-72982effba51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:37.166437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5515c5-7415-4c09-9704-72982effba51 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:37.166916 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5515c5-7415-4c09-9704-72982effba51 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:37.167339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5515c5-7415-4c09-9704-72982effba51 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:37.168228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5515c5-7415-4c09-9704-72982effba51 None admin] RBAC: Adding query filter params (group.id=f3b2710f65d74a5a8180706d20789222, role.id=22819414415942d399db534708fe7103, scope.project.id=3dcdfc529b6149688ceee6b949a735a1) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:37.168696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5515c5-7415-4c09-9704-72982effba51 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:37.170317 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5515c5-7415-4c09-9704-72982effba51 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:37.177082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 195/380] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Sep 13 15:16:36 2022] GET /identity/v3/role_assignments?group.id=f3b2710f65d74a5a8180706d20789222&scope.project.id=3dcdfc529b6149688ceee6b949a735a1&role.id=22819414415942d399db534708fe7103 => generated 260 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:37.581929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00beeaf1-587f-427c-81a2-7f69c8cf18a4 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:37.582511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00beeaf1-587f-427c-81a2-7f69c8cf18a4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:37.582511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00beeaf1-587f-427c-81a2-7f69c8cf18a4 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:37.583988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 186/381] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:37.589408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a521463-51ee-4aa2-aa1d-97e6db7b50a8 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:37.590222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a521463-51ee-4aa2-aa1d-97e6db7b50a8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:37.590704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a521463-51ee-4aa2-aa1d-97e6db7b50a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:37.611938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-2a521463-51ee-4aa2-aa1d-97e6db7b50a8 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:37.613447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2a521463-51ee-4aa2-aa1d-97e6db7b50a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:37.713669 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 196/382] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:38.000290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00beeaf1-587f-427c-81a2-7f69c8cf18a4 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:38.114197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:38.143787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00beeaf1-587f-427c-81a2-7f69c8cf18a4 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:38.188301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee16a7e-ed3b-447f-bb1e-d798d5dd9967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:38.188930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee16a7e-ed3b-447f-bb1e-d798d5dd9967 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:38.189204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee16a7e-ed3b-447f-bb1e-d798d5dd9967 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:38.189469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee16a7e-ed3b-447f-bb1e-d798d5dd9967 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:38.202932 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee16a7e-ed3b-447f-bb1e-d798d5dd9967 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-cee16a7e-ed3b-447f-bb1e-d798d5dd9967 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:38.207814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:38.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:38.209568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 187/383] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:38.214982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a521463-51ee-4aa2-aa1d-97e6db7b50a8 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:38.284096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:38.335275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a521463-51ee-4aa2-aa1d-97e6db7b50a8 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:38.383777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8f8e1e-9151-4882-85cb-a76385f62023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:38.384394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8f8e1e-9151-4882-85cb-a76385f62023 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:38.384670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8f8e1e-9151-4882-85cb-a76385f62023 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:38.384936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8f8e1e-9151-4882-85cb-a76385f62023 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:38.385605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8f8e1e-9151-4882-85cb-a76385f62023 None admin] RBAC: Adding query filter params (name=member) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:38.385907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8f8e1e-9151-4882-85cb-a76385f62023 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:38.387301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8f8e1e-9151-4882-85cb-a76385f62023 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:38.402736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 197/384] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:38.408498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee16a7e-ed3b-447f-bb1e-d798d5dd9967 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:38.563676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee16a7e-ed3b-447f-bb1e-d798d5dd9967 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:38.611721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883f3edf-4d61-4ae3-8eed-9929120ee9b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:38.612376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883f3edf-4d61-4ae3-8eed-9929120ee9b9 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:38.612621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883f3edf-4d61-4ae3-8eed-9929120ee9b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:38.612880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883f3edf-4d61-4ae3-8eed-9929120ee9b9 None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:38.614338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883f3edf-4d61-4ae3-8eed-9929120ee9b9 None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:38.615790 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883f3edf-4d61-4ae3-8eed-9929120ee9b9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:38.617360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 188/385] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:38 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:38.623067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8f8e1e-9151-4882-85cb-a76385f62023 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:38.784880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8f8e1e-9151-4882-85cb-a76385f62023 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:38.817520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:38.836102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589dc003-af8f-4e5c-a830-2059c10df736 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:38.836516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589dc003-af8f-4e5c-a830-2059c10df736 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:38.836850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589dc003-af8f-4e5c-a830-2059c10df736 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:38.839024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceb78fc-7911-4c07-b012-0540cc41994f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:38.839904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceb78fc-7911-4c07-b012-0540cc41994f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:38.840383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceb78fc-7911-4c07-b012-0540cc41994f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:38.840856 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceb78fc-7911-4c07-b012-0540cc41994f None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:38.843823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 189/386] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:38.844721 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ceb78fc-7911-4c07-b012-0540cc41994f None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:38.846562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ceb78fc-7911-4c07-b012-0540cc41994f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:38.850275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922a3d9c-7bae-41d0-bea0-2bfa13328e2c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:38.850688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922a3d9c-7bae-41d0-bea0-2bfa13328e2c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:38.851076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922a3d9c-7bae-41d0-bea0-2bfa13328e2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:38.853393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 198/387] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:38 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:38.854607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceb78fc-7911-4c07-b012-0540cc41994f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:38.888916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-922a3d9c-7bae-41d0-bea0-2bfa13328e2c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:38.890162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-922a3d9c-7bae-41d0-bea0-2bfa13328e2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:38.904057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:39.021551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 190/388] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:39.030178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceb78fc-7911-4c07-b012-0540cc41994f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:39.081545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1178bd6-5bac-4811-9b0a-df88543bd80c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:39.082423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1178bd6-5bac-4811-9b0a-df88543bd80c None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:39.082729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1178bd6-5bac-4811-9b0a-df88543bd80c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:39.083004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1178bd6-5bac-4811-9b0a-df88543bd80c None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/groups/69bbd4b43fc64649997434589044be4c/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:39.085179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1178bd6-5bac-4811-9b0a-df88543bd80c None admin] RBAC: Authorizing `identity:create_grant(project_id=1cf34cdf97b64f59bf465ae56dd45517, group_id=69bbd4b43fc64649997434589044be4c, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:39.086906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1178bd6-5bac-4811-9b0a-df88543bd80c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:39.117319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 199/389] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Sep 13 15:16:38 2022] PUT /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/groups/69bbd4b43fc64649997434589044be4c/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:39.192395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8826a-7675-453a-ac36-58a5413d0363 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:39.193042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8826a-7675-453a-ac36-58a5413d0363 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:39.193550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e8826a-7675-453a-ac36-58a5413d0363 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:39.195570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 191/390] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:39.205564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d482f1-3e16-4077-a6a2-18f0f8c5938c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:39.205891 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d482f1-3e16-4077-a6a2-18f0f8c5938c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:39.206125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d482f1-3e16-4077-a6a2-18f0f8c5938c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:39.229048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a0d482f1-3e16-4077-a6a2-18f0f8c5938c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:39.230690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a0d482f1-3e16-4077-a6a2-18f0f8c5938c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:39.293458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:39.321419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 200/391] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:39.351580 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e8826a-7675-453a-ac36-58a5413d0363 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:39.488304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e8826a-7675-453a-ac36-58a5413d0363 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:39.501082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf381f47-efda-45a4-8d69-2b8eda1f1cd7 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:39.501375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf381f47-efda-45a4-8d69-2b8eda1f1cd7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:39.501617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf381f47-efda-45a4-8d69-2b8eda1f1cd7 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:39.503272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 201/392] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:16:39.515131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31651fc7-1ed7-4da2-a398-d390228e7483 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:39.515422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31651fc7-1ed7-4da2-a398-d390228e7483 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:39.515674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31651fc7-1ed7-4da2-a398-d390228e7483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:39.527317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8acd8bd-91d3-4659-a89d-0d8ed69f2f21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:39.528305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8acd8bd-91d3-4659-a89d-0d8ed69f2f21 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:39.528780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8acd8bd-91d3-4659-a89d-0d8ed69f2f21 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:39.529343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8acd8bd-91d3-4659-a89d-0d8ed69f2f21 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:39.535736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-31651fc7-1ed7-4da2-a398-d390228e7483 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:39.537089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-31651fc7-1ed7-4da2-a398-d390228e7483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:39.540478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:39.544596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8acd8bd-91d3-4659-a89d-0d8ed69f2f21 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-f8acd8bd-91d3-4659-a89d-0d8ed69f2f21 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:39.550785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:39.551667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:39.552433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 192/393] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:39.558502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8acd8bd-91d3-4659-a89d-0d8ed69f2f21 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:39.652095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 202/394] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:39.653913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31651fc7-1ed7-4da2-a398-d390228e7483 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:39.739539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8acd8bd-91d3-4659-a89d-0d8ed69f2f21 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:39.760571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31651fc7-1ed7-4da2-a398-d390228e7483 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:39.771284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:39.798232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07da34c-813f-4f89-96b2-16e249b93082 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:39.799022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07da34c-813f-4f89-96b2-16e249b93082 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:39.799301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07da34c-813f-4f89-96b2-16e249b93082 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:39.799572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07da34c-813f-4f89-96b2-16e249b93082 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:39.810325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07da34c-813f-4f89-96b2-16e249b93082 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:39.811901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c4e433-7dd5-4610-9c57-cb2dc099331d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:39.812541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c4e433-7dd5-4610-9c57-cb2dc099331d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:39.812821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c4e433-7dd5-4610-9c57-cb2dc099331d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:39.813099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c4e433-7dd5-4610-9c57-cb2dc099331d None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:39.813780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c4e433-7dd5-4610-9c57-cb2dc099331d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:39.814083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c4e433-7dd5-4610-9c57-cb2dc099331d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:39.815498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c4e433-7dd5-4610-9c57-cb2dc099331d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-f07da34c-813f-4f89-96b2-16e249b93082 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:39.815600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:39.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:39.817157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 203/395] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:39.824589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 193/396] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:39.825601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07da34c-813f-4f89-96b2-16e249b93082 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:39.832624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c4e433-7dd5-4610-9c57-cb2dc099331d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:39.948057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:40.016717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c4e433-7dd5-4610-9c57-cb2dc099331d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.022897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07da34c-813f-4f89-96b2-16e249b93082 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.053708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96af482-4c41-49d3-a168-1c28a9c31783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.054402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96af482-4c41-49d3-a168-1c28a9c31783 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.054681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96af482-4c41-49d3-a168-1c28a9c31783 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.054959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96af482-4c41-49d3-a168-1c28a9c31783 None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.056563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96af482-4c41-49d3-a168-1c28a9c31783 None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:40.057869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96af482-4c41-49d3-a168-1c28a9c31783 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:40.060109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 194/397] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:39 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:40.061139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96af482-4c41-49d3-a168-1c28a9c31783 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:40.066234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62771c46-5104-4d88-86ab-3f99075fcb1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.067255 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62771c46-5104-4d88-86ab-3f99075fcb1f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.067741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62771c46-5104-4d88-86ab-3f99075fcb1f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.068214 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62771c46-5104-4d88-86ab-3f99075fcb1f None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.069099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62771c46-5104-4d88-86ab-3f99075fcb1f None admin] RBAC: Adding query filter params (name=member) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:40.069598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62771c46-5104-4d88-86ab-3f99075fcb1f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:40.071364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62771c46-5104-4d88-86ab-3f99075fcb1f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:40.082765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 204/398] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:40.084185 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62771c46-5104-4d88-86ab-3f99075fcb1f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:40.119926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:40.174235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96af482-4c41-49d3-a168-1c28a9c31783 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.204501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62771c46-5104-4d88-86ab-3f99075fcb1f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.208319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:40.214332 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3fb746-586c-4055-a37a-6bc1817bafd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.214976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3fb746-586c-4055-a37a-6bc1817bafd6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.215254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3fb746-586c-4055-a37a-6bc1817bafd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.215522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3fb746-586c-4055-a37a-6bc1817bafd6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.226223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3fb746-586c-4055-a37a-6bc1817bafd6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-ce3fb746-586c-4055-a37a-6bc1817bafd6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:40.231501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:40.232385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:40.233242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 195/399] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:40.233885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3fb746-586c-4055-a37a-6bc1817bafd6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:40.245791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396338ea-08ae-47f1-b4da-4eafa329d7f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.246467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396338ea-08ae-47f1-b4da-4eafa329d7f0 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.246745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396338ea-08ae-47f1-b4da-4eafa329d7f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.247039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396338ea-08ae-47f1-b4da-4eafa329d7f0 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.248176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396338ea-08ae-47f1-b4da-4eafa329d7f0 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:40.249697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396338ea-08ae-47f1-b4da-4eafa329d7f0 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:40.256313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 205/400] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:40 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:40.256313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f834bd6c-0d6e-4de1-9ef1-872c643747a9 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.256313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f834bd6c-0d6e-4de1-9ef1-872c643747a9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.256313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f834bd6c-0d6e-4de1-9ef1-872c643747a9 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.256313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 206/401] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:40.256313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f834bd6c-0d6e-4de1-9ef1-872c643747a9 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:40.348446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3fb746-586c-4055-a37a-6bc1817bafd6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.376303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:40.382747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f834bd6c-0d6e-4de1-9ef1-872c643747a9 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.387906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7800f3-7744-4b07-b3b7-82b5687def38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.388547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7800f3-7744-4b07-b3b7-82b5687def38 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.388820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7800f3-7744-4b07-b3b7-82b5687def38 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.389087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7800f3-7744-4b07-b3b7-82b5687def38 None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.390286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7800f3-7744-4b07-b3b7-82b5687def38 None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:40.391743 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7800f3-7744-4b07-b3b7-82b5687def38 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:40.397978 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 196/402] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:40 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:40.397978 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7800f3-7744-4b07-b3b7-82b5687def38 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:40.420144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011fe60b-22bf-4908-a472-a7fa26b8ada5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.420862 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fe60b-22bf-4908-a472-a7fa26b8ada5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.421252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fe60b-22bf-4908-a472-a7fa26b8ada5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.421563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fe60b-22bf-4908-a472-a7fa26b8ada5 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.422257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011fe60b-22bf-4908-a472-a7fa26b8ada5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:40.422565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011fe60b-22bf-4908-a472-a7fa26b8ada5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:40.423938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011fe60b-22bf-4908-a472-a7fa26b8ada5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:40.432230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 207/403] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:40.433251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9cecc4-0052-4323-80c3-e3c1d2c04115 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.433536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9cecc4-0052-4323-80c3-e3c1d2c04115 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.433806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9cecc4-0052-4323-80c3-e3c1d2c04115 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.452767 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-cf9cecc4-0052-4323-80c3-e3c1d2c04115 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:40.453965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-cf9cecc4-0052-4323-80c3-e3c1d2c04115 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:40.460825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:40.507546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7800f3-7744-4b07-b3b7-82b5687def38 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.524897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 208/404] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:40.525498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9cecc4-0052-4323-80c3-e3c1d2c04115 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:40.549079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b4256d-47b2-4680-a978-0a76223d30a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.550133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b4256d-47b2-4680-a978-0a76223d30a7 None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.550636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b4256d-47b2-4680-a978-0a76223d30a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.551611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b4256d-47b2-4680-a978-0a76223d30a7 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/groups/69bbd4b43fc64649997434589044be4c/roles/74db8755b1f04d839260ccd3e36deb66` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.554524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b4256d-47b2-4680-a978-0a76223d30a7 None admin] RBAC: Authorizing `identity:create_grant(project_id=1cf34cdf97b64f59bf465ae56dd45517, group_id=69bbd4b43fc64649997434589044be4c, role_id=74db8755b1f04d839260ccd3e36deb66)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:40.557085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b4256d-47b2-4680-a978-0a76223d30a7 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:40.579742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 197/405] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Sep 13 15:16:40 2022] PUT /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517/groups/69bbd4b43fc64649997434589044be4c/roles/74db8755b1f04d839260ccd3e36deb66 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:40.580887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b4256d-47b2-4680-a978-0a76223d30a7 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:40.671029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:40.694593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9cecc4-0052-4323-80c3-e3c1d2c04115 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.706709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b4256d-47b2-4680-a978-0a76223d30a7 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.726808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:40.749215 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dccf534-7cfd-4d45-b940-753ab3d17a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.749855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dccf534-7cfd-4d45-b940-753ab3d17a66 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.750160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dccf534-7cfd-4d45-b940-753ab3d17a66 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.750438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dccf534-7cfd-4d45-b940-753ab3d17a66 None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.753287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dccf534-7cfd-4d45-b940-753ab3d17a66 None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:40.753621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dccf534-7cfd-4d45-b940-753ab3d17a66 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:40.756238 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 198/406] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:40 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:40.764695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dccf534-7cfd-4d45-b940-753ab3d17a66 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:40.777296 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565704ee-6253-4170-bb09-47c69978c285 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.777993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565704ee-6253-4170-bb09-47c69978c285 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.778304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565704ee-6253-4170-bb09-47c69978c285 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.778568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565704ee-6253-4170-bb09-47c69978c285 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.782023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565704ee-6253-4170-bb09-47c69978c285 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:40.783560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565704ee-6253-4170-bb09-47c69978c285 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:40.785707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 209/407] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:40 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:40.791388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565704ee-6253-4170-bb09-47c69978c285 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:40.895118 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dccf534-7cfd-4d45-b940-753ab3d17a66 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.924045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565704ee-6253-4170-bb09-47c69978c285 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:40.937861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a649319-e88d-4852-a4bf-676256026784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.938890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a649319-e88d-4852-a4bf-676256026784 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.939383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a649319-e88d-4852-a4bf-676256026784 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.939972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a649319-e88d-4852-a4bf-676256026784 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.941379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a649319-e88d-4852-a4bf-676256026784 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:40.943169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a649319-e88d-4852-a4bf-676256026784 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:40.945587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 199/408] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:40 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:40.946718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a649319-e88d-4852-a4bf-676256026784 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:40.966753 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0ef380-0358-41d1-9084-499d8945d757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:40.967828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0ef380-0358-41d1-9084-499d8945d757 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:40.968333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0ef380-0358-41d1-9084-499d8945d757 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:40.969054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0ef380-0358-41d1-9084-499d8945d757 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19/groups/69bbd4b43fc64649997434589044be4c/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:40.971235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0ef380-0358-41d1-9084-499d8945d757 None admin] RBAC: Authorizing `identity:create_grant(project_id=dea71e3f98bf423598a93def5ac1cd19, group_id=69bbd4b43fc64649997434589044be4c, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:40.972997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa0ef380-0358-41d1-9084-499d8945d757 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:40.993483 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 210/409] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Sep 13 15:16:40 2022] PUT /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19/groups/69bbd4b43fc64649997434589044be4c/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:16:40.999522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0ef380-0358-41d1-9084-499d8945d757 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:41.030448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:41.040368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:41.110738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a649319-e88d-4852-a4bf-676256026784 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:41.153826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52adcd90-fb3d-49c0-9fa2-4b8e0b844484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:41.156104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52adcd90-fb3d-49c0-9fa2-4b8e0b844484 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:41.162659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52adcd90-fb3d-49c0-9fa2-4b8e0b844484 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:41.163236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52adcd90-fb3d-49c0-9fa2-4b8e0b844484 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:41.177127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52adcd90-fb3d-49c0-9fa2-4b8e0b844484 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-52adcd90-fb3d-49c0-9fa2-4b8e0b844484 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:41.184671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:41.185634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:41.186879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:41.190049 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 200/410] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:16:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:41.197211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52adcd90-fb3d-49c0-9fa2-4b8e0b844484 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:41.248365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0ef380-0358-41d1-9084-499d8945d757 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:41.286968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30ece95-4450-45bc-9740-7218897a81b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:41.287655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30ece95-4450-45bc-9740-7218897a81b4 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:41.287933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30ece95-4450-45bc-9740-7218897a81b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:41.288207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30ece95-4450-45bc-9740-7218897a81b4 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19/groups/69bbd4b43fc64649997434589044be4c/roles/74db8755b1f04d839260ccd3e36deb66` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:41.290489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30ece95-4450-45bc-9740-7218897a81b4 None admin] RBAC: Authorizing `identity:create_grant(project_id=dea71e3f98bf423598a93def5ac1cd19, group_id=69bbd4b43fc64649997434589044be4c, role_id=74db8755b1f04d839260ccd3e36deb66)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:41.291946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30ece95-4450-45bc-9740-7218897a81b4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:41.311712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 211/411] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Sep 13 15:16:40 2022] PUT /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19/groups/69bbd4b43fc64649997434589044be4c/roles/74db8755b1f04d839260ccd3e36deb66 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:16:41.313177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52adcd90-fb3d-49c0-9fa2-4b8e0b844484 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:41.338161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:41.363415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5dfdb3-b07b-4025-a437-b667b96309b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:41.364711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5dfdb3-b07b-4025-a437-b667b96309b8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:41.365163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5dfdb3-b07b-4025-a437-b667b96309b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:41.365580 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5dfdb3-b07b-4025-a437-b667b96309b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:41.366436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5dfdb3-b07b-4025-a437-b667b96309b8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:41.367413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5dfdb3-b07b-4025-a437-b667b96309b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:41.368738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5dfdb3-b07b-4025-a437-b667b96309b8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:41.379904 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 201/412] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:41.387382 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30ece95-4450-45bc-9740-7218897a81b4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:41.494446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:41.520837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30ece95-4450-45bc-9740-7218897a81b4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:41.557412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eafb2f8-a298-4947-b7ae-07e2a01b5858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:41.558371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eafb2f8-a298-4947-b7ae-07e2a01b5858 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:41.558788 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eafb2f8-a298-4947-b7ae-07e2a01b5858 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:41.559198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eafb2f8-a298-4947-b7ae-07e2a01b5858 None admin] PATH_INFO: `/v3/groups/f3b2710f65d74a5a8180706d20789222` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:41.560560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eafb2f8-a298-4947-b7ae-07e2a01b5858 None admin] RBAC: Authorizing `identity:get_group(group_id=f3b2710f65d74a5a8180706d20789222)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:41.562262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eafb2f8-a298-4947-b7ae-07e2a01b5858 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:41.564445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 212/413] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:41 2022] GET /identity/v3/groups/f3b2710f65d74a5a8180706d20789222 => generated 229 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:41.570288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5dfdb3-b07b-4025-a437-b667b96309b8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:41.711651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5dfdb3-b07b-4025-a437-b667b96309b8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:41.752841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80fc10-dcc4-4516-b33a-cd0609ac780b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:41.754176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80fc10-dcc4-4516-b33a-cd0609ac780b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:41.754663 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80fc10-dcc4-4516-b33a-cd0609ac780b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:41.755094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80fc10-dcc4-4516-b33a-cd0609ac780b None admin] PATH_INFO: `/v3/projects/3dcdfc529b6149688ceee6b949a735a1` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:41.756402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80fc10-dcc4-4516-b33a-cd0609ac780b None admin] RBAC: Authorizing `identity:get_project(project_id=3dcdfc529b6149688ceee6b949a735a1)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:41.758010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80fc10-dcc4-4516-b33a-cd0609ac780b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:41.760100 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 202/414] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:41 2022] GET /identity/v3/projects/3dcdfc529b6149688ceee6b949a735a1 => generated 344 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:41.766312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eafb2f8-a298-4947-b7ae-07e2a01b5858 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:41.831377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb07bf-6fc4-45bd-a106-f57a7bdea2d9 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:41.831550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb07bf-6fc4-45bd-a106-f57a7bdea2d9 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:41.831828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfb07bf-6fc4-45bd-a106-f57a7bdea2d9 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:41.833438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 203/415] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:16:41.838417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b044cb-8ce0-44a1-976a-682d5ddb3b4e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:41.838697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b044cb-8ce0-44a1-976a-682d5ddb3b4e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:41.838966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b044cb-8ce0-44a1-976a-682d5ddb3b4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:41.857688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-a9b044cb-8ce0-44a1-976a-682d5ddb3b4e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:41.859133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a9b044cb-8ce0-44a1-976a-682d5ddb3b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:41.922451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:41.932754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eafb2f8-a298-4947-b7ae-07e2a01b5858 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:41.978083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 204/416] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:41.999261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d048ab77-40d7-4f5a-97a6-37f9ea72231c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:41.999943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d048ab77-40d7-4f5a-97a6-37f9ea72231c None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:42.000216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d048ab77-40d7-4f5a-97a6-37f9ea72231c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:42.000486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d048ab77-40d7-4f5a-97a6-37f9ea72231c None admin] PATH_INFO: `/v3/projects/3dcdfc529b6149688ceee6b949a735a1/groups/f3b2710f65d74a5a8180706d20789222/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:42.007427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d048ab77-40d7-4f5a-97a6-37f9ea72231c None admin] RBAC: Authorizing `identity:create_grant(project_id=3dcdfc529b6149688ceee6b949a735a1, group_id=f3b2710f65d74a5a8180706d20789222, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:42.008932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d048ab77-40d7-4f5a-97a6-37f9ea72231c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:42.056926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 213/417] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Sep 13 15:16:41 2022] PUT /identity/v3/projects/3dcdfc529b6149688ceee6b949a735a1/groups/f3b2710f65d74a5a8180706d20789222/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:16:42.317275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b044cb-8ce0-44a1-976a-682d5ddb3b4e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:42.436969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b044cb-8ce0-44a1-976a-682d5ddb3b4e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:42.478755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798c12b0-f002-4543-b42f-ccf39d0e5ae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:42.479401 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798c12b0-f002-4543-b42f-ccf39d0e5ae4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:42.479682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798c12b0-f002-4543-b42f-ccf39d0e5ae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:42.479959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798c12b0-f002-4543-b42f-ccf39d0e5ae4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:42.498211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798c12b0-f002-4543-b42f-ccf39d0e5ae4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-798c12b0-f002-4543-b42f-ccf39d0e5ae4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:42.503094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:42.504643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:42.506587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 205/418] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:42.516957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d048ab77-40d7-4f5a-97a6-37f9ea72231c None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:42.528482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:42.665735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d048ab77-40d7-4f5a-97a6-37f9ea72231c None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:42.698714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee5b1f0-c3f7-41d8-8173-8fbc69e2d493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:42.699373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee5b1f0-c3f7-41d8-8173-8fbc69e2d493 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:42.699650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee5b1f0-c3f7-41d8-8173-8fbc69e2d493 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:42.699918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee5b1f0-c3f7-41d8-8173-8fbc69e2d493 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:42.700578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee5b1f0-c3f7-41d8-8173-8fbc69e2d493 None admin] RBAC: Adding query filter params (name=member) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:42.700877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee5b1f0-c3f7-41d8-8173-8fbc69e2d493 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:42.702381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee5b1f0-c3f7-41d8-8173-8fbc69e2d493 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:42.716709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 214/419] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:42.718297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798c12b0-f002-4543-b42f-ccf39d0e5ae4 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:42.774438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:42.860261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798c12b0-f002-4543-b42f-ccf39d0e5ae4 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:42.914779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40966947-209e-4244-8aaa-65e2146b4785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:42.915757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40966947-209e-4244-8aaa-65e2146b4785 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:42.916233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40966947-209e-4244-8aaa-65e2146b4785 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:42.916695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40966947-209e-4244-8aaa-65e2146b4785 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:42.992502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40966947-209e-4244-8aaa-65e2146b4785 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:42.994354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40966947-209e-4244-8aaa-65e2146b4785 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:43.027354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:43.100833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 206/420] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:43.106354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee5b1f0-c3f7-41d8-8173-8fbc69e2d493 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:43.218874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:43.268229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee5b1f0-c3f7-41d8-8173-8fbc69e2d493 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:43.319863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a7b470-7c77-4827-94c5-3e627123b013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:43.320514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a7b470-7c77-4827-94c5-3e627123b013 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:43.320790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a7b470-7c77-4827-94c5-3e627123b013 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:43.321061 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a7b470-7c77-4827-94c5-3e627123b013 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:43.326783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a7b470-7c77-4827-94c5-3e627123b013 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:43.328584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a7b470-7c77-4827-94c5-3e627123b013 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:43.330587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 215/421] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:43 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:43.336072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40966947-209e-4244-8aaa-65e2146b4785 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:43.338887 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5780c57e-c1cb-47a7-b657-dfb8bae2a27b None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:43.339461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5780c57e-c1cb-47a7-b657-dfb8bae2a27b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:43.339927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5780c57e-c1cb-47a7-b657-dfb8bae2a27b None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:43.341616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 216/422] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:16:43.346835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a7e5ce-0f0a-452f-b411-bf0c639c1c05 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:43.347373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a7e5ce-0f0a-452f-b411-bf0c639c1c05 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:43.347827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a7e5ce-0f0a-452f-b411-bf0c639c1c05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:43.387596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-21a7e5ce-0f0a-452f-b411-bf0c639c1c05 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:43.389113 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-21a7e5ce-0f0a-452f-b411-bf0c639c1c05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:43.508433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40966947-209e-4244-8aaa-65e2146b4785 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:43.514541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 217/423] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:43.547846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211b58e5-1621-4643-8127-6fa2265452cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:43.548870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211b58e5-1621-4643-8127-6fa2265452cc None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:43.549336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211b58e5-1621-4643-8127-6fa2265452cc None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:43.549787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211b58e5-1621-4643-8127-6fa2265452cc None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:43.551138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211b58e5-1621-4643-8127-6fa2265452cc None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:43.552779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211b58e5-1621-4643-8127-6fa2265452cc None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:43.554897 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 207/424] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:43 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:43.563271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a7e5ce-0f0a-452f-b411-bf0c639c1c05 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:43.651933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:43.693763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a7e5ce-0f0a-452f-b411-bf0c639c1c05 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:43.728783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bd86b5-bb9f-4bf8-b993-894fde00fbe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:43.729393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd86b5-bb9f-4bf8-b993-894fde00fbe2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:43.729661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd86b5-bb9f-4bf8-b993-894fde00fbe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:43.730028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd86b5-bb9f-4bf8-b993-894fde00fbe2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:43.730776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bd86b5-bb9f-4bf8-b993-894fde00fbe2 None admin] RBAC: Adding query filter params (group.id=69bbd4b43fc64649997434589044be4c, role.id=7a7e816668a7493bad0752baca37934c, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:43.731076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bd86b5-bb9f-4bf8-b993-894fde00fbe2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:43.732542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57bd86b5-bb9f-4bf8-b993-894fde00fbe2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:43.738681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 218/425] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Sep 13 15:16:43 2022] GET /identity/v3/role_assignments?group.id=69bbd4b43fc64649997434589044be4c&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=7a7e816668a7493bad0752baca37934c => generated 611 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:43.810080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c01a57-ea8e-43fe-b08b-0c7ff4215e86 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:43.810610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c01a57-ea8e-43fe-b08b-0c7ff4215e86 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:43.810610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c01a57-ea8e-43fe-b08b-0c7ff4215e86 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:43.812321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 208/426] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:43.820051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de39e8bf-3a85-41b7-b527-ffccc6b145ee None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:43.820329 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de39e8bf-3a85-41b7-b527-ffccc6b145ee None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:43.820597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de39e8bf-3a85-41b7-b527-ffccc6b145ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:43.838800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c01a57-ea8e-43fe-b08b-0c7ff4215e86 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:43.845882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:43.858916 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-de39e8bf-3a85-41b7-b527-ffccc6b145ee None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:43.860157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-de39e8bf-3a85-41b7-b527-ffccc6b145ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:43.977267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c01a57-ea8e-43fe-b08b-0c7ff4215e86 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:44.004813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 219/427] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:16:44.017411 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2ddbba-1020-4857-b3d2-57cddb1a549d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:44.018055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2ddbba-1020-4857-b3d2-57cddb1a549d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.018356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2ddbba-1020-4857-b3d2-57cddb1a549d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.018627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2ddbba-1020-4857-b3d2-57cddb1a549d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.029464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2ddbba-1020-4857-b3d2-57cddb1a549d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-fe2ddbba-1020-4857-b3d2-57cddb1a549d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:44.034425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:44.035297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:44.036345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 209/428] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:44.041655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de39e8bf-3a85-41b7-b527-ffccc6b145ee None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:44.107282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:44.150852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de39e8bf-3a85-41b7-b527-ffccc6b145ee None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:44.188737 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2890d3-2211-4c36-a2ad-b57a2f0e459e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:44.189362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2890d3-2211-4c36-a2ad-b57a2f0e459e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.189638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2890d3-2211-4c36-a2ad-b57a2f0e459e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.189959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2890d3-2211-4c36-a2ad-b57a2f0e459e None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.190660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2890d3-2211-4c36-a2ad-b57a2f0e459e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:44.190998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2890d3-2211-4c36-a2ad-b57a2f0e459e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:44.192372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2890d3-2211-4c36-a2ad-b57a2f0e459e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:44.201911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 220/429] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:44.207905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2ddbba-1020-4857-b3d2-57cddb1a549d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:44.282719 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:44.317354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2ddbba-1020-4857-b3d2-57cddb1a549d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:44.338899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2890d3-2211-4c36-a2ad-b57a2f0e459e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:44.369971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e67dd6-bbde-4263-b420-45dc12c5b1ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:44.370830 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e67dd6-bbde-4263-b420-45dc12c5b1ce None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.371112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e67dd6-bbde-4263-b420-45dc12c5b1ce None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.371804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e67dd6-bbde-4263-b420-45dc12c5b1ce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.456628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e67dd6-bbde-4263-b420-45dc12c5b1ce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:44.458718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e67dd6-bbde-4263-b420-45dc12c5b1ce None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:44.461065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2890d3-2211-4c36-a2ad-b57a2f0e459e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:44.486556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:44.506424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-218abc59-2656-4efe-a2c7-8f2213bde1fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:44.507316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218abc59-2656-4efe-a2c7-8f2213bde1fc None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.507758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218abc59-2656-4efe-a2c7-8f2213bde1fc None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.508191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-218abc59-2656-4efe-a2c7-8f2213bde1fc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.524166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-218abc59-2656-4efe-a2c7-8f2213bde1fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-218abc59-2656-4efe-a2c7-8f2213bde1fc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:44.529490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:44.530461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 13 15:16:44.531288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:44.532363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:44.538084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 221/430] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:16:44.538084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6720680b-7679-4fcc-b01a-733cb51687fd None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.538084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6720680b-7679-4fcc-b01a-733cb51687fd None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.538084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6720680b-7679-4fcc-b01a-733cb51687fd None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.538084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 222/431] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:44.543344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6720680b-7679-4fcc-b01a-733cb51687fd None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:44.596045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 210/432] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:44.596045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501add9-8361-47b4-aeaa-fac70d951d4b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.596045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501add9-8361-47b4-aeaa-fac70d951d4b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.596045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d501add9-8361-47b4-aeaa-fac70d951d4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.612095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d501add9-8361-47b4-aeaa-fac70d951d4b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:44.613290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d501add9-8361-47b4-aeaa-fac70d951d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:44.671062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6720680b-7679-4fcc-b01a-733cb51687fd None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:44.684467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 211/433] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:44.685414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d501add9-8361-47b4-aeaa-fac70d951d4b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:44.715318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce3e3b7-157c-4d6f-abe4-fd41d8caa906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:44.715909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce3e3b7-157c-4d6f-abe4-fd41d8caa906 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.716219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce3e3b7-157c-4d6f-abe4-fd41d8caa906 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.716536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce3e3b7-157c-4d6f-abe4-fd41d8caa906 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.717312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce3e3b7-157c-4d6f-abe4-fd41d8caa906 None admin] RBAC: Adding query filter params (name=member) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:44.718090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce3e3b7-157c-4d6f-abe4-fd41d8caa906 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:44.719248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce3e3b7-157c-4d6f-abe4-fd41d8caa906 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:44.732604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 223/434] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:16:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:44.732604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a73033-1563-4526-9118-5d5c69fb433d None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.732604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a73033-1563-4526-9118-5d5c69fb433d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.732604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a73033-1563-4526-9118-5d5c69fb433d None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.732604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 224/435] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:44 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:44.733876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a73033-1563-4526-9118-5d5c69fb433d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:44.783053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:44.794176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:44.803545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d501add9-8361-47b4-aeaa-fac70d951d4b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:44.837673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf650cb-f59e-4086-a285-2284508bb0c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:44.838317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf650cb-f59e-4086-a285-2284508bb0c0 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.838592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf650cb-f59e-4086-a285-2284508bb0c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.838866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf650cb-f59e-4086-a285-2284508bb0c0 None admin] PATH_INFO: `/v3/projects/1cf34cdf97b64f59bf465ae56dd45517` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.840284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf650cb-f59e-4086-a285-2284508bb0c0 None admin] RBAC: Authorizing `identity:get_project(project_id=1cf34cdf97b64f59bf465ae56dd45517)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:44.841729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf650cb-f59e-4086-a285-2284508bb0c0 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:44.848263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 212/436] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:44 2022] GET /identity/v3/projects/1cf34cdf97b64f59bf465ae56dd45517 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:44.848419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a73033-1563-4526-9118-5d5c69fb433d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:44.848500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dc0e87-34a8-4ade-863b-8b4059dde821 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.848500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dc0e87-34a8-4ade-863b-8b4059dde821 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.848500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dc0e87-34a8-4ade-863b-8b4059dde821 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.864823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e6dc0e87-34a8-4ade-863b-8b4059dde821 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:44.866293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e6dc0e87-34a8-4ade-863b-8b4059dde821 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:44.883819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569cbe9a-8ddb-427b-ac5c-a93b62780cae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:44.884431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569cbe9a-8ddb-427b-ac5c-a93b62780cae None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:44.884704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569cbe9a-8ddb-427b-ac5c-a93b62780cae None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:44.884971 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569cbe9a-8ddb-427b-ac5c-a93b62780cae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:44.942133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 213/437] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:44.942133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dc0e87-34a8-4ade-863b-8b4059dde821 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:44.969075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569cbe9a-8ddb-427b-ac5c-a93b62780cae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:44.970676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569cbe9a-8ddb-427b-ac5c-a93b62780cae None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:45.021238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:45.029699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:45.041153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dc0e87-34a8-4ade-863b-8b4059dde821 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.076504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 225/438] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:45.077251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569cbe9a-8ddb-427b-ac5c-a93b62780cae None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:45.080895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65687ec4-0a96-491d-aef2-57d4188751d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.081602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65687ec4-0a96-491d-aef2-57d4188751d8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.081885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65687ec4-0a96-491d-aef2-57d4188751d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.082178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65687ec4-0a96-491d-aef2-57d4188751d8 None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.083417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65687ec4-0a96-491d-aef2-57d4188751d8 None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:45.084859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65687ec4-0a96-491d-aef2-57d4188751d8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:45.091520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 214/439] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:44 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:45.092333 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65687ec4-0a96-491d-aef2-57d4188751d8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:45.182406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569cbe9a-8ddb-427b-ac5c-a93b62780cae None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.197195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65687ec4-0a96-491d-aef2-57d4188751d8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.216604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f55e3e7-b65f-4893-85ba-14ed344336f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.217254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f55e3e7-b65f-4893-85ba-14ed344336f5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.217518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f55e3e7-b65f-4893-85ba-14ed344336f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.217787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f55e3e7-b65f-4893-85ba-14ed344336f5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.227662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f55e3e7-b65f-4893-85ba-14ed344336f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:45.230656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e0a052-be39-4b23-8a65-a3124c7acca0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.231287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e0a052-be39-4b23-8a65-a3124c7acca0 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.231564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e0a052-be39-4b23-8a65-a3124c7acca0 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.231835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e0a052-be39-4b23-8a65-a3124c7acca0 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-3f55e3e7-b65f-4893-85ba-14ed344336f5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:45.232468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:45.233657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:45.234495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:45.235194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e0a052-be39-4b23-8a65-a3124c7acca0 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:45.235264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 226/440] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:45.235264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f55e3e7-b65f-4893-85ba-14ed344336f5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:45.235378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e0a052-be39-4b23-8a65-a3124c7acca0 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:45.236644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 215/441] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:45 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:45.237392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e0a052-be39-4b23-8a65-a3124c7acca0 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:45.245497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:45.266393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:45.339884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e0a052-be39-4b23-8a65-a3124c7acca0 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.340118 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f55e3e7-b65f-4893-85ba-14ed344336f5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.374273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6183a8-2500-416f-8dfa-93b335d49010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.374894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6183a8-2500-416f-8dfa-93b335d49010 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.375162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6183a8-2500-416f-8dfa-93b335d49010 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.375428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6183a8-2500-416f-8dfa-93b335d49010 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.376570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feaf9d25-4442-43b1-8f9e-3acd3ec31256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.377226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feaf9d25-4442-43b1-8f9e-3acd3ec31256 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.377507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feaf9d25-4442-43b1-8f9e-3acd3ec31256 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.377765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feaf9d25-4442-43b1-8f9e-3acd3ec31256 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.378519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feaf9d25-4442-43b1-8f9e-3acd3ec31256 None admin] RBAC: Adding query filter params (group.id=69bbd4b43fc64649997434589044be4c, role.id=74db8755b1f04d839260ccd3e36deb66, scope.project.id=1cf34cdf97b64f59bf465ae56dd45517) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:45.378829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feaf9d25-4442-43b1-8f9e-3acd3ec31256 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:45.380287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feaf9d25-4442-43b1-8f9e-3acd3ec31256 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:45.385136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6183a8-2500-416f-8dfa-93b335d49010 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:45.385959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 227/442] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Sep 13 15:16:45 2022] GET /identity/v3/role_assignments?group.id=69bbd4b43fc64649997434589044be4c&scope.project.id=1cf34cdf97b64f59bf465ae56dd45517&role.id=74db8755b1f04d839260ccd3e36deb66 => generated 611 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:45.386759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feaf9d25-4442-43b1-8f9e-3acd3ec31256 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-8f6183a8-2500-416f-8dfa-93b335d49010 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:45.390443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:45.391579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:45.392385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 216/443] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:16:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:45.393361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6183a8-2500-416f-8dfa-93b335d49010 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:45.478125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:45.485313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:45.502561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6183a8-2500-416f-8dfa-93b335d49010 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.526698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feaf9d25-4442-43b1-8f9e-3acd3ec31256 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.540174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b948e543-f2ce-4b04-9c48-d8de79bd8887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.540944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b948e543-f2ce-4b04-9c48-d8de79bd8887 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.541224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b948e543-f2ce-4b04-9c48-d8de79bd8887 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.541497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b948e543-f2ce-4b04-9c48-d8de79bd8887 None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.543455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b948e543-f2ce-4b04-9c48-d8de79bd8887 None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:45.545493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b948e543-f2ce-4b04-9c48-d8de79bd8887 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:45.552756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 217/444] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:45 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:45.552756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b948e543-f2ce-4b04-9c48-d8de79bd8887 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:45.563985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2931d1f4-fb49-42b3-969f-eb17d51a3d14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.564632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2931d1f4-fb49-42b3-969f-eb17d51a3d14 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.565448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2931d1f4-fb49-42b3-969f-eb17d51a3d14 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.565448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2931d1f4-fb49-42b3-969f-eb17d51a3d14 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.565839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2931d1f4-fb49-42b3-969f-eb17d51a3d14 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:45.566152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2931d1f4-fb49-42b3-969f-eb17d51a3d14 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:45.567590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2931d1f4-fb49-42b3-969f-eb17d51a3d14 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:45.580921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 228/445] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:45.581855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2931d1f4-fb49-42b3-969f-eb17d51a3d14 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:45.691216 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b948e543-f2ce-4b04-9c48-d8de79bd8887 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.699907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2931d1f4-fb49-42b3-969f-eb17d51a3d14 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.731143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50086644-07ad-4aa6-bf2a-dbaa990689d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.731707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50086644-07ad-4aa6-bf2a-dbaa990689d6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.731983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50086644-07ad-4aa6-bf2a-dbaa990689d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.732249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50086644-07ad-4aa6-bf2a-dbaa990689d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.732910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50086644-07ad-4aa6-bf2a-dbaa990689d6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:45.733240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50086644-07ad-4aa6-bf2a-dbaa990689d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:45.734082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:45.734646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50086644-07ad-4aa6-bf2a-dbaa990689d6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:45.742640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2900340-aad4-46fa-8e24-f4583769ab9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.743250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2900340-aad4-46fa-8e24-f4583769ab9d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.743517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2900340-aad4-46fa-8e24-f4583769ab9d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.743781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2900340-aad4-46fa-8e24-f4583769ab9d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.744501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2900340-aad4-46fa-8e24-f4583769ab9d None admin] RBAC: Adding query filter params (group.id=69bbd4b43fc64649997434589044be4c, role.id=7a7e816668a7493bad0752baca37934c, scope.project.id=dea71e3f98bf423598a93def5ac1cd19) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:45.744803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2900340-aad4-46fa-8e24-f4583769ab9d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:45.745836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 218/446] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:45.746163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50086644-07ad-4aa6-bf2a-dbaa990689d6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:45.746259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2900340-aad4-46fa-8e24-f4583769ab9d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:45.752595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 229/447] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Sep 13 15:16:45 2022] GET /identity/v3/role_assignments?group.id=69bbd4b43fc64649997434589044be4c&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=7a7e816668a7493bad0752baca37934c => generated 611 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:45.760947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2900340-aad4-46fa-8e24-f4583769ab9d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:45.768805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:45.876919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50086644-07ad-4aa6-bf2a-dbaa990689d6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.880726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2900340-aad4-46fa-8e24-f4583769ab9d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:45.914090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddea674-d1d3-4e0b-9d3f-f25ddb8ede8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.914863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddea674-d1d3-4e0b-9d3f-f25ddb8ede8c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.915208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddea674-d1d3-4e0b-9d3f-f25ddb8ede8c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.915545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddea674-d1d3-4e0b-9d3f-f25ddb8ede8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.918385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a0005b-a08f-402e-9c57-b3c7bada2578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:45.918921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a0005b-a08f-402e-9c57-b3c7bada2578 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:45.919198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a0005b-a08f-402e-9c57-b3c7bada2578 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:45.919461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a0005b-a08f-402e-9c57-b3c7bada2578 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:45.992008 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:45.993520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddea674-d1d3-4e0b-9d3f-f25ddb8ede8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:45.996033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddea674-d1d3-4e0b-9d3f-f25ddb8ede8c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:46.001291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a0005b-a08f-402e-9c57-b3c7bada2578 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:46.001599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a0005b-a08f-402e-9c57-b3c7bada2578 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:46.016345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:46.114683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 219/448] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:46.119543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 230/449] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:46.121366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddea674-d1d3-4e0b-9d3f-f25ddb8ede8c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:46.126229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a0005b-a08f-402e-9c57-b3c7bada2578 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:46.217383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:46.221953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddea674-d1d3-4e0b-9d3f-f25ddb8ede8c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:46.224718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a0005b-a08f-402e-9c57-b3c7bada2578 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:46.238296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:46.256338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d42d75-23c6-48cf-acca-3c8525c5d621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:46.256892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d42d75-23c6-48cf-acca-3c8525c5d621 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:46.257165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d42d75-23c6-48cf-acca-3c8525c5d621 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:46.257429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d42d75-23c6-48cf-acca-3c8525c5d621 None admin] PATH_INFO: `/v3/projects/dea71e3f98bf423598a93def5ac1cd19` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:46.258533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d42d75-23c6-48cf-acca-3c8525c5d621 None admin] RBAC: Authorizing `identity:get_project(project_id=dea71e3f98bf423598a93def5ac1cd19)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:46.259287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb838502-11eb-4075-a796-dd29a7c7cb10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:46.259927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb838502-11eb-4075-a796-dd29a7c7cb10 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:46.260022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d42d75-23c6-48cf-acca-3c8525c5d621 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:46.260203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb838502-11eb-4075-a796-dd29a7c7cb10 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:46.260472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb838502-11eb-4075-a796-dd29a7c7cb10 None admin] PATH_INFO: `/v3/projects/3dcdfc529b6149688ceee6b949a735a1` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:46.261542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 220/450] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:46 2022] GET /identity/v3/projects/dea71e3f98bf423598a93def5ac1cd19 => generated 302 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:46.265913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb838502-11eb-4075-a796-dd29a7c7cb10 None admin] RBAC: Authorizing `identity:get_project(project_id=3dcdfc529b6149688ceee6b949a735a1)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:46.265913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb838502-11eb-4075-a796-dd29a7c7cb10 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:46.265913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 231/451] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:16:46 2022] GET /identity/v3/projects/3dcdfc529b6149688ceee6b949a735a1 => generated 344 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:46.267477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d42d75-23c6-48cf-acca-3c8525c5d621 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:46.270334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb838502-11eb-4075-a796-dd29a7c7cb10 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:46.364840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d42d75-23c6-48cf-acca-3c8525c5d621 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:46.368649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb838502-11eb-4075-a796-dd29a7c7cb10 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:46.400158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5d8b6d-ede7-4ac3-91ff-3a2ae665fd68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:46.400804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5d8b6d-ede7-4ac3-91ff-3a2ae665fd68 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:46.401082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5d8b6d-ede7-4ac3-91ff-3a2ae665fd68 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:46.401354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5d8b6d-ede7-4ac3-91ff-3a2ae665fd68 None admin] PATH_INFO: `/v3/groups/69bbd4b43fc64649997434589044be4c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:46.402156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f474dfaa-2cb3-4716-98f5-09db27d38b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:46.402479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5d8b6d-ede7-4ac3-91ff-3a2ae665fd68 None admin] RBAC: Authorizing `identity:get_group(group_id=69bbd4b43fc64649997434589044be4c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:46.402782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f474dfaa-2cb3-4716-98f5-09db27d38b30 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:46.403056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f474dfaa-2cb3-4716-98f5-09db27d38b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:46.403326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f474dfaa-2cb3-4716-98f5-09db27d38b30 None admin] PATH_INFO: `/v3/groups/f3b2710f65d74a5a8180706d20789222` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:46.403931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5d8b6d-ede7-4ac3-91ff-3a2ae665fd68 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:46.404497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f474dfaa-2cb3-4716-98f5-09db27d38b30 None admin] RBAC: Authorizing `identity:get_group(group_id=f3b2710f65d74a5a8180706d20789222)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:46.405711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 221/452] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:46 2022] GET /identity/v3/groups/69bbd4b43fc64649997434589044be4c => generated 226 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:46.406014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f474dfaa-2cb3-4716-98f5-09db27d38b30 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:46.413172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 232/453] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Sep 13 15:16:46 2022] GET /identity/v3/groups/f3b2710f65d74a5a8180706d20789222 => generated 229 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:46.413969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f474dfaa-2cb3-4716-98f5-09db27d38b30 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:46.414145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5d8b6d-ede7-4ac3-91ff-3a2ae665fd68 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:46.444962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:46.469661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:46.515945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f474dfaa-2cb3-4716-98f5-09db27d38b30 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:46.517767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5d8b6d-ede7-4ac3-91ff-3a2ae665fd68 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:46.548779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014c146f-7746-4340-a6c9-03759ca08d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:46.549379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c146f-7746-4340-a6c9-03759ca08d5b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:46.549647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c146f-7746-4340-a6c9-03759ca08d5b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:46.549911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c146f-7746-4340-a6c9-03759ca08d5b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:46.550658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014c146f-7746-4340-a6c9-03759ca08d5b None admin] RBAC: Adding query filter params (group.id=f3b2710f65d74a5a8180706d20789222, role.id=22819414415942d399db534708fe7103, scope.project.id=3dcdfc529b6149688ceee6b949a735a1) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:46.550959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014c146f-7746-4340-a6c9-03759ca08d5b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:46.552367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014c146f-7746-4340-a6c9-03759ca08d5b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:46.553649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc72b1-4e39-4d04-8a2b-2adec1c4da18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:46.554309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc72b1-4e39-4d04-8a2b-2adec1c4da18 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:46.554587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc72b1-4e39-4d04-8a2b-2adec1c4da18 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:46.554856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc72b1-4e39-4d04-8a2b-2adec1c4da18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:46.555581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fc72b1-4e39-4d04-8a2b-2adec1c4da18 None admin] RBAC: Adding query filter params (group.id=69bbd4b43fc64649997434589044be4c, role.id=74db8755b1f04d839260ccd3e36deb66, scope.project.id=dea71e3f98bf423598a93def5ac1cd19) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:46.555887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fc72b1-4e39-4d04-8a2b-2adec1c4da18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:46.557303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fc72b1-4e39-4d04-8a2b-2adec1c4da18 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:46.558050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 233/454] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Sep 13 15:16:46 2022] GET /identity/v3/role_assignments?group.id=f3b2710f65d74a5a8180706d20789222&scope.project.id=3dcdfc529b6149688ceee6b949a735a1&role.id=22819414415942d399db534708fe7103 => generated 611 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:46.564144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 222/455] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Sep 13 15:16:46 2022] GET /identity/v3/role_assignments?group.id=69bbd4b43fc64649997434589044be4c&scope.project.id=dea71e3f98bf423598a93def5ac1cd19&role.id=74db8755b1f04d839260ccd3e36deb66 => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:49.278473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6011819d-b232-4e1a-a2ef-6b3712b3161e None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:49.279001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6011819d-b232-4e1a-a2ef-6b3712b3161e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:49.279303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6011819d-b232-4e1a-a2ef-6b3712b3161e None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:49.281298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 234/456] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:49.286722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cf55e6-ede5-439b-8320-b14e6718942f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:49.287183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cf55e6-ede5-439b-8320-b14e6718942f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:49.287259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cf55e6-ede5-439b-8320-b14e6718942f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:49.312690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-09cf55e6-ede5-439b-8320-b14e6718942f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:49.313907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-09cf55e6-ede5-439b-8320-b14e6718942f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:49.404722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 223/457] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:49.717495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6011819d-b232-4e1a-a2ef-6b3712b3161e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:49.830707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:49.843078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6011819d-b232-4e1a-a2ef-6b3712b3161e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:49.856015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268e6344-cd20-4ba5-8fda-8d149ff915c5 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:49.856340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268e6344-cd20-4ba5-8fda-8d149ff915c5 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:49.856562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268e6344-cd20-4ba5-8fda-8d149ff915c5 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:49.859736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 224/458] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:49.864629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646dabbb-5dfa-4f2c-8c8e-e865fe2df407 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:49.865165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646dabbb-5dfa-4f2c-8c8e-e865fe2df407 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:49.865624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646dabbb-5dfa-4f2c-8c8e-e865fe2df407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:49.892899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a33e95-7b46-415d-baeb-2b668c1c9c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:49.893547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a33e95-7b46-415d-baeb-2b668c1c9c35 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:49.893821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a33e95-7b46-415d-baeb-2b668c1c9c35 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:49.893931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-646dabbb-5dfa-4f2c-8c8e-e865fe2df407 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:49.894178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a33e95-7b46-415d-baeb-2b668c1c9c35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:49.895156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-646dabbb-5dfa-4f2c-8c8e-e865fe2df407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:49.906820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a33e95-7b46-415d-baeb-2b668c1c9c35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:49.909217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a33e95-7b46-415d-baeb-2b668c1c9c35 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-24a33e95-7b46-415d-baeb-2b668c1c9c35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:49.931174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:49.932150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:49.933005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:49.933005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:49.933005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:49.933005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 235/459] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:49.938127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77b50e2-7ba4-40bf-af57-febdc9d28fe3 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:49.938127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77b50e2-7ba4-40bf-af57-febdc9d28fe3 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:49.938127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77b50e2-7ba4-40bf-af57-febdc9d28fe3 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:49.938127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 236/460] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:49.939140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77b50e2-7ba4-40bf-af57-febdc9d28fe3 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:49.952779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:49.979957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 225/461] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:49.981511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa35f785-571f-4cb3-a2aa-98e1cea3734a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:49.981797 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa35f785-571f-4cb3-a2aa-98e1cea3734a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:49.982073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa35f785-571f-4cb3-a2aa-98e1cea3734a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:50.008317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-aa35f785-571f-4cb3-a2aa-98e1cea3734a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:50.010064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-aa35f785-571f-4cb3-a2aa-98e1cea3734a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:50.061385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77b50e2-7ba4-40bf-af57-febdc9d28fe3 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:50.092425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 226/462] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:50.094321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19cb7db-1b05-4c26-b1d7-d56622cca31f None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:50.094607 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19cb7db-1b05-4c26-b1d7-d56622cca31f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:50.094973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19cb7db-1b05-4c26-b1d7-d56622cca31f None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:50.096276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f623f160-222f-4f5e-b618-44d6f2fd60b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:50.096629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 227/463] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:50.101312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623f160-222f-4f5e-b618-44d6f2fd60b5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:50.101596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623f160-222f-4f5e-b618-44d6f2fd60b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:50.101869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623f160-222f-4f5e-b618-44d6f2fd60b5 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:50.102553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f623f160-222f-4f5e-b618-44d6f2fd60b5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:50.102865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f623f160-222f-4f5e-b618-44d6f2fd60b5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:50.103251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c31323c-af70-48af-bb63-c354e39666a0 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:50.103532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c31323c-af70-48af-bb63-c354e39666a0 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:50.103805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c31323c-af70-48af-bb63-c354e39666a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:50.104261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f623f160-222f-4f5e-b618-44d6f2fd60b5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:50.123293 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 237/464] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:50.134125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f623f160-222f-4f5e-b618-44d6f2fd60b5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:50.142483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-5c31323c-af70-48af-bb63-c354e39666a0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:50.142783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:50.145533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-5c31323c-af70-48af-bb63-c354e39666a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:50.261633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 228/465] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:50.288027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f623f160-222f-4f5e-b618-44d6f2fd60b5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:50.323439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c31323c-af70-48af-bb63-c354e39666a0 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:50.330000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a96ae71-c93c-4db2-92d1-03393d7def57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:50.330653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a96ae71-c93c-4db2-92d1-03393d7def57 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:50.330928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a96ae71-c93c-4db2-92d1-03393d7def57 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:50.331195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a96ae71-c93c-4db2-92d1-03393d7def57 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:50.331853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a96ae71-c93c-4db2-92d1-03393d7def57 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:50.333285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a96ae71-c93c-4db2-92d1-03393d7def57 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:50.362986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:50.380469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 238/466] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:16:50 2022] POST /identity/v3/users => generated 246 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:16:50.382354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a96ae71-c93c-4db2-92d1-03393d7def57 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:50.498046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c31323c-af70-48af-bb63-c354e39666a0 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:50.521095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:50.571034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d959c568-02ab-4dad-9322-1c9b7f26a7ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:50.571702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959c568-02ab-4dad-9322-1c9b7f26a7ab None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:50.571982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959c568-02ab-4dad-9322-1c9b7f26a7ab None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:50.572255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d959c568-02ab-4dad-9322-1c9b7f26a7ab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:50.586427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d959c568-02ab-4dad-9322-1c9b7f26a7ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:50.587906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d959c568-02ab-4dad-9322-1c9b7f26a7ab None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:50.588006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a96ae71-c93c-4db2-92d1-03393d7def57 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-d959c568-02ab-4dad-9322-1c9b7f26a7ab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:50.604803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:50.605747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:50.606670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:50.606670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:50.606670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:50.608220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 229/467] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:50.609164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d959c568-02ab-4dad-9322-1c9b7f26a7ab None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:50.632988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2a4d38-a33c-4498-8c00-f9000a914115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:50.633647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a4d38-a33c-4498-8c00-f9000a914115 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:50.633916 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a4d38-a33c-4498-8c00-f9000a914115 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:50.634224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a4d38-a33c-4498-8c00-f9000a914115 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:50.654272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2a4d38-a33c-4498-8c00-f9000a914115 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:50.656131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2a4d38-a33c-4498-8c00-f9000a914115 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-ff2a4d38-a33c-4498-8c00-f9000a914115 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:50.675301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:50.676245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:50.677108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:50.677108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:50.677108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:50.677108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 239/468] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:50.677518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2a4d38-a33c-4498-8c00-f9000a914115 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:50.716365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:50.730903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d959c568-02ab-4dad-9322-1c9b7f26a7ab None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:50.783340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dd7da1-caf0-4d9d-b256-a85a298aa87e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:50.784258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dd7da1-caf0-4d9d-b256-a85a298aa87e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:50.784704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dd7da1-caf0-4d9d-b256-a85a298aa87e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:50.785116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dd7da1-caf0-4d9d-b256-a85a298aa87e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:50.797597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3dd7da1-caf0-4d9d-b256-a85a298aa87e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:50.799351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3dd7da1-caf0-4d9d-b256-a85a298aa87e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:50.812796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2a4d38-a33c-4498-8c00-f9000a914115 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-c3dd7da1-caf0-4d9d-b256-a85a298aa87e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:50.815866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:50.816753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:50.817663 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:50.817663 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:50.817663 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:50.819755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 230/469] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:50.820796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dd7da1-caf0-4d9d-b256-a85a298aa87e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:50.835898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:50.860875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a2c059-f819-48e5-ac18-d7a2ab703b64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:50.861899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a2c059-f819-48e5-ac18-d7a2ab703b64 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:50.862380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a2c059-f819-48e5-ac18-d7a2ab703b64 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:50.862821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a2c059-f819-48e5-ac18-d7a2ab703b64 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:50.863667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a2c059-f819-48e5-ac18-d7a2ab703b64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:50.864480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a2c059-f819-48e5-ac18-d7a2ab703b64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:50.866095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a2c059-f819-48e5-ac18-d7a2ab703b64 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:50.879632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 240/470] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:50.880844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a2c059-f819-48e5-ac18-d7a2ab703b64 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:50.938834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dd7da1-caf0-4d9d-b256-a85a298aa87e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:50.974790 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b35cf3-2af1-4bad-8899-85703d927236 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:50.975433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b35cf3-2af1-4bad-8899-85703d927236 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:50.975718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b35cf3-2af1-4bad-8899-85703d927236 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:50.975989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b35cf3-2af1-4bad-8899-85703d927236 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:50.976635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b35cf3-2af1-4bad-8899-85703d927236 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:50.976937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b35cf3-2af1-4bad-8899-85703d927236 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:50.978365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b35cf3-2af1-4bad-8899-85703d927236 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:50.986555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a2c059-f819-48e5-ac18-d7a2ab703b64 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:50.987640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:50.993639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 231/471] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:50.998602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b35cf3-2af1-4bad-8899-85703d927236 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:51.026691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b98fc55-9b9a-43ad-8983-42b1ca6d483e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:51.027307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b98fc55-9b9a-43ad-8983-42b1ca6d483e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:51.027580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b98fc55-9b9a-43ad-8983-42b1ca6d483e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:51.027848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b98fc55-9b9a-43ad-8983-42b1ca6d483e None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:51.028490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b98fc55-9b9a-43ad-8983-42b1ca6d483e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:51.028786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b98fc55-9b9a-43ad-8983-42b1ca6d483e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:51.030268 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b98fc55-9b9a-43ad-8983-42b1ca6d483e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:51.047290 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 241/472] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:51.047290 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fb852-2d26-4772-9603-143ab9b4f7e1 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:51.047290 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fb852-2d26-4772-9603-143ab9b4f7e1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:51.047290 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fb852-2d26-4772-9603-143ab9b4f7e1 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:51.047290 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 242/473] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:51 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:51.047290 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fb852-2d26-4772-9603-143ab9b4f7e1 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:51.083034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:51.109837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b35cf3-2af1-4bad-8899-85703d927236 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:51.145911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aa0ab8-2e81-481c-99ca-13775f2f4754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:51.146548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aa0ab8-2e81-481c-99ca-13775f2f4754 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:51.146805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aa0ab8-2e81-481c-99ca-13775f2f4754 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:51.147073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aa0ab8-2e81-481c-99ca-13775f2f4754 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:51.147748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1aa0ab8-2e81-481c-99ca-13775f2f4754 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:51.149203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1aa0ab8-2e81-481c-99ca-13775f2f4754 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:51.154559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fb852-2d26-4772-9603-143ab9b4f7e1 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:51.178146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 232/474] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:16:50 2022] POST /identity/v3/users => generated 248 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:16:51.178146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aa0ab8-2e81-481c-99ca-13775f2f4754 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:51.201619 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa91ec94-3877-4cbb-b0a3-d831ea65ed22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:51.202486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa91ec94-3877-4cbb-b0a3-d831ea65ed22 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:51.202833 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa91ec94-3877-4cbb-b0a3-d831ea65ed22 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:51.203169 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa91ec94-3877-4cbb-b0a3-d831ea65ed22 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:51.204033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa91ec94-3877-4cbb-b0a3-d831ea65ed22 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:51.205689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa91ec94-3877-4cbb-b0a3-d831ea65ed22 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:51.237634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 243/475] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:16:51 2022] POST /identity/v3/users => generated 248 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:16:51.237634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c0e40-ed79-4d94-bddd-dc36c051b999 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:51.237634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c0e40-ed79-4d94-bddd-dc36c051b999 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:51.237634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c0e40-ed79-4d94-bddd-dc36c051b999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:51.269657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-896c0e40-ed79-4d94-bddd-dc36c051b999 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:51.271243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-896c0e40-ed79-4d94-bddd-dc36c051b999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:51.305538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:51.416244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aa0ab8-2e81-481c-99ca-13775f2f4754 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:51.422040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 244/476] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:51.454901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e31dd3-d8e0-46fe-b5b9-0fdfb81438b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:51.455503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e31dd3-d8e0-46fe-b5b9-0fdfb81438b5 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:51.455777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e31dd3-d8e0-46fe-b5b9-0fdfb81438b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:51.456042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e31dd3-d8e0-46fe-b5b9-0fdfb81438b5 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:51.456690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e31dd3-d8e0-46fe-b5b9-0fdfb81438b5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:51.458210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e31dd3-d8e0-46fe-b5b9-0fdfb81438b5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:51.486181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 233/477] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:16:51 2022] POST /identity/v3/users => generated 249 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:16:51.700972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896c0e40-ed79-4d94-bddd-dc36c051b999 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:51.727850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:51.819284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896c0e40-ed79-4d94-bddd-dc36c051b999 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:51.865890 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823f7ee2-95e0-485e-816c-27fae690177e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:51.866565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823f7ee2-95e0-485e-816c-27fae690177e None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:51.866847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823f7ee2-95e0-485e-816c-27fae690177e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:51.867120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823f7ee2-95e0-485e-816c-27fae690177e None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:51.867809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823f7ee2-95e0-485e-816c-27fae690177e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:51.869346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823f7ee2-95e0-485e-816c-27fae690177e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:51.888005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.sql.core [None req-823f7ee2-95e0-485e-816c-27fae690177e None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 13 15:16:51.888005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 13 15:16:51.888005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [parameters: {'id': '607ad87809324099a1716d8caa495b7c', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 13 15:16:51.888005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81455) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-823f7ee2-95e0-485e-816c-27fae690177e None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application result.read() Sep 13 15:16:51.904419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return ref.to_dict() Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:16:51.905508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application yield resource Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self.session.rollback() Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self.force_reraise() Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise self.value Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application session.commit() Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 13 15:16:51.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._prepare_impl() Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self.session.flush() Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._flush(objects) Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application compat.raise_( Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application flush_context.execute() Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rec.execute(self) Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 13 15:16:51.907440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:16:51.908301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application result.read() Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:16:51.909101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [parameters: {'id': '607ad87809324099a1716d8caa495b7c', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:51.909857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 13 15:16:51.912579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:51.914544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 245/478] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Sep 13 15:16:51 2022] POST /identity/v3/roles => generated 144 bytes in 206 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:51.920664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e31dd3-d8e0-46fe-b5b9-0fdfb81438b5 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:52.056775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e31dd3-d8e0-46fe-b5b9-0fdfb81438b5 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:52.072799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:52.101233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b653663-db3a-4a87-9ca8-de4cd66f97dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:52.101883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b653663-db3a-4a87-9ca8-de4cd66f97dd None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:52.102180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b653663-db3a-4a87-9ca8-de4cd66f97dd None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:52.102454 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b653663-db3a-4a87-9ca8-de4cd66f97dd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:52.112828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b653663-db3a-4a87-9ca8-de4cd66f97dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-6b653663-db3a-4a87-9ca8-de4cd66f97dd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:52.118344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:52.119184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 13 15:16:52.119986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:52.122054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 234/479] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:16:51 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:52.127398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823f7ee2-95e0-485e-816c-27fae690177e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:52.242696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:52.263688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823f7ee2-95e0-485e-816c-27fae690177e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:52.309444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc59fd41-c392-49ea-a581-9786e488f010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:52.310071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc59fd41-c392-49ea-a581-9786e488f010 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:52.310371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc59fd41-c392-49ea-a581-9786e488f010 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:52.310644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc59fd41-c392-49ea-a581-9786e488f010 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:52.311413 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc59fd41-c392-49ea-a581-9786e488f010 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:52.311697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc59fd41-c392-49ea-a581-9786e488f010 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:52.313274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc59fd41-c392-49ea-a581-9786e488f010 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:52.322464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 246/480] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:52.855960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d6b3ae-640c-4896-a6b2-45a75d23e19e None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:52.856984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d6b3ae-640c-4896-a6b2-45a75d23e19e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:52.857463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d6b3ae-640c-4896-a6b2-45a75d23e19e None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:52.859667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 235/481] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:52.866366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0a9b90-4c74-4311-98df-a8a84d4baec0 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:52.867217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0a9b90-4c74-4311-98df-a8a84d4baec0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:52.867674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0a9b90-4c74-4311-98df-a8a84d4baec0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:52.886065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-1e0a9b90-4c74-4311-98df-a8a84d4baec0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:52.887562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1e0a9b90-4c74-4311-98df-a8a84d4baec0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:52.955830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 247/482] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:53.301505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d6b3ae-640c-4896-a6b2-45a75d23e19e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:53.407230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d6b3ae-640c-4896-a6b2-45a75d23e19e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:53.463690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba1e7fb-6bec-4657-a6dc-7385f9c553a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:53.464879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba1e7fb-6bec-4657-a6dc-7385f9c553a7 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:53.465347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba1e7fb-6bec-4657-a6dc-7385f9c553a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:53.465827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba1e7fb-6bec-4657-a6dc-7385f9c553a7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:53.480848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba1e7fb-6bec-4657-a6dc-7385f9c553a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-fba1e7fb-6bec-4657-a6dc-7385f9c553a7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:53.486502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:53.487375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:16:53.488186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:53.491233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 236/483] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:16:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:53.497278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0a9b90-4c74-4311-98df-a8a84d4baec0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:53.591410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:53.615715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0a9b90-4c74-4311-98df-a8a84d4baec0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:53.665836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e78f48-d665-4457-96b6-5bce7363aa6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:53.666474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e78f48-d665-4457-96b6-5bce7363aa6e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:53.666745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e78f48-d665-4457-96b6-5bce7363aa6e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:53.667007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e78f48-d665-4457-96b6-5bce7363aa6e None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:53.667670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e78f48-d665-4457-96b6-5bce7363aa6e None admin] RBAC: Adding query filter params (name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:53.667973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e78f48-d665-4457-96b6-5bce7363aa6e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:53.669354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e78f48-d665-4457-96b6-5bce7363aa6e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:53.681896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 248/484] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:16:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:53.688124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba1e7fb-6bec-4657-a6dc-7385f9c553a7 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:53.731032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:53.813085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba1e7fb-6bec-4657-a6dc-7385f9c553a7 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:53.859846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2695805-7a35-4247-8009-a858ca1bd2b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:53.860456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2695805-7a35-4247-8009-a858ca1bd2b4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:53.860701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2695805-7a35-4247-8009-a858ca1bd2b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:53.860925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2695805-7a35-4247-8009-a858ca1bd2b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:53.871946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2695805-7a35-4247-8009-a858ca1bd2b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:53.873493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2695805-7a35-4247-8009-a858ca1bd2b4 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:53.887691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b5e55e-8dff-499a-ba2e-9cc36f3747cb None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:53.888205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b5e55e-8dff-499a-ba2e-9cc36f3747cb None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:53.888205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b5e55e-8dff-499a-ba2e-9cc36f3747cb None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:53.889747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 249/485] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-b2695805-7a35-4247-8009-a858ca1bd2b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:53.891162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:53.892018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:53.892840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:53.892840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:53.892840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:53.892840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 237/486] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:53.897685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f700733-cf70-4b0f-873d-e59b1c15de3e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:53.897962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f700733-cf70-4b0f-873d-e59b1c15de3e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:53.898256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f700733-cf70-4b0f-873d-e59b1c15de3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:53.903743 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2695805-7a35-4247-8009-a858ca1bd2b4 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:53.921110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8f700733-cf70-4b0f-873d-e59b1c15de3e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:53.922354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8f700733-cf70-4b0f-873d-e59b1c15de3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:54.048196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 250/487] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:54.054068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:54.072552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2695805-7a35-4247-8009-a858ca1bd2b4 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:54.107464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cf135a-616c-4471-acbc-1634b28b674e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:54.108132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cf135a-616c-4471-acbc-1634b28b674e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.108466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cf135a-616c-4471-acbc-1634b28b674e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.108737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cf135a-616c-4471-acbc-1634b28b674e None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.109378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cf135a-616c-4471-acbc-1634b28b674e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:54.109675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cf135a-616c-4471-acbc-1634b28b674e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:54.111143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cf135a-616c-4471-acbc-1634b28b674e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:54.127620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 238/488] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:54.133199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f700733-cf70-4b0f-873d-e59b1c15de3e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:54.155221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89056a0f-86fc-4f88-a0c0-33728b09fffa None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.155516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89056a0f-86fc-4f88-a0c0-33728b09fffa None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.155760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89056a0f-86fc-4f88-a0c0-33728b09fffa None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.157380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 239/489] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:16:54.165711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80bc347-091d-4f31-abd7-025b36ab3081 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.165988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80bc347-091d-4f31-abd7-025b36ab3081 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.166286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80bc347-091d-4f31-abd7-025b36ab3081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.184569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c80bc347-091d-4f31-abd7-025b36ab3081 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:54.185772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c80bc347-091d-4f31-abd7-025b36ab3081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:54.218865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:54.260494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f700733-cf70-4b0f-873d-e59b1c15de3e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:54.271802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 240/490] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:54.273427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8ad911-2141-4dce-a2da-38e1ddbbe1f4 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.273917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8ad911-2141-4dce-a2da-38e1ddbbe1f4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.274410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8ad911-2141-4dce-a2da-38e1ddbbe1f4 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.280746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 241/491] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:54.282620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a65f1a-76a9-43cc-8354-61c0c2ccaaef None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.283125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a65f1a-76a9-43cc-8354-61c0c2ccaaef None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.283596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a65f1a-76a9-43cc-8354-61c0c2ccaaef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.303245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-58a65f1a-76a9-43cc-8354-61c0c2ccaaef None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:54.304998 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-58a65f1a-76a9-43cc-8354-61c0c2ccaaef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:54.334880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008835f4-f0a9-4f4a-9d39-3e3f127b9b3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:54.335527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008835f4-f0a9-4f4a-9d39-3e3f127b9b3b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.335805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008835f4-f0a9-4f4a-9d39-3e3f127b9b3b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.336075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008835f4-f0a9-4f4a-9d39-3e3f127b9b3b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.344832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008835f4-f0a9-4f4a-9d39-3e3f127b9b3b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:54.346310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008835f4-f0a9-4f4a-9d39-3e3f127b9b3b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-008835f4-f0a9-4f4a-9d39-3e3f127b9b3b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:16:54.362593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:54.364018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 251/492] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Sep 13 15:16:54 2022] GET /identity/v3/users/nova => generated 82 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:54.365259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008835f4-f0a9-4f4a-9d39-3e3f127b9b3b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:54.387793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 242/493] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:54.388893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a65f1a-76a9-43cc-8354-61c0c2ccaaef None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:54.397229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:54.516299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008835f4-f0a9-4f4a-9d39-3e3f127b9b3b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:54.543031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:54.549430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a65f1a-76a9-43cc-8354-61c0c2ccaaef None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:54.552274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad327a8e-9536-437e-b838-49969376ce47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:54.553723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad327a8e-9536-437e-b838-49969376ce47 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.554008 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad327a8e-9536-437e-b838-49969376ce47 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.554314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad327a8e-9536-437e-b838-49969376ce47 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.566289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad327a8e-9536-437e-b838-49969376ce47 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-ad327a8e-9536-437e-b838-49969376ce47 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:54.572153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:54.573046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:54.573822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 252/494] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:16:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:54.580200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad327a8e-9536-437e-b838-49969376ce47 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:54.630922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29629b7-7c5d-4802-b4ba-7c85c383b757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:54.631471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29629b7-7c5d-4802-b4ba-7c85c383b757 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.631745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29629b7-7c5d-4802-b4ba-7c85c383b757 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.632011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29629b7-7c5d-4802-b4ba-7c85c383b757 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.632753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29629b7-7c5d-4802-b4ba-7c85c383b757 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:54.633054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29629b7-7c5d-4802-b4ba-7c85c383b757 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:54.634537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29629b7-7c5d-4802-b4ba-7c85c383b757 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:54.682490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 243/495] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Sep 13 15:16:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:54.683458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29629b7-7c5d-4802-b4ba-7c85c383b757 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:54.700546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad327a8e-9536-437e-b838-49969376ce47 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:54.737113 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eedfc08-05c1-4289-bede-fffa00b16652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:54.738193 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eedfc08-05c1-4289-bede-fffa00b16652 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.738487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eedfc08-05c1-4289-bede-fffa00b16652 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.739196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eedfc08-05c1-4289-bede-fffa00b16652 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.740022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eedfc08-05c1-4289-bede-fffa00b16652 None admin] RBAC: Adding query filter params (name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:54.740333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eedfc08-05c1-4289-bede-fffa00b16652 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:54.742284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eedfc08-05c1-4289-bede-fffa00b16652 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:54.746851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:54.750994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 253/496] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:16:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:54.751812 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eedfc08-05c1-4289-bede-fffa00b16652 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:54.791503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29629b7-7c5d-4802-b4ba-7c85c383b757 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:54.806287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:54.824956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440a213e-9651-457f-9322-3da497514e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:54.825587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a213e-9651-457f-9322-3da497514e3e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.825864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a213e-9651-457f-9322-3da497514e3e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.826150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a213e-9651-457f-9322-3da497514e3e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.836253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440a213e-9651-457f-9322-3da497514e3e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-440a213e-9651-457f-9322-3da497514e3e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:54.840951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:54.841809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 244/497] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:16:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:54.843012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440a213e-9651-457f-9322-3da497514e3e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:54.866845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eedfc08-05c1-4289-bede-fffa00b16652 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:54.901847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fd71dc-b79c-4b9d-82f0-47be536e9d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:54.902516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fd71dc-b79c-4b9d-82f0-47be536e9d2a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.902700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fd71dc-b79c-4b9d-82f0-47be536e9d2a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.902968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fd71dc-b79c-4b9d-82f0-47be536e9d2a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.946393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440a213e-9651-457f-9322-3da497514e3e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:54.968575 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:54.973567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fd71dc-b79c-4b9d-82f0-47be536e9d2a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:54.975050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fd71dc-b79c-4b9d-82f0-47be536e9d2a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:54.980670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdb9607-e63d-4ac9-906a-77799dee22d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:54.981307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdb9607-e63d-4ac9-906a-77799dee22d6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:54.981590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdb9607-e63d-4ac9-906a-77799dee22d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:54.981861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdb9607-e63d-4ac9-906a-77799dee22d6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:54.992545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdb9607-e63d-4ac9-906a-77799dee22d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-1bdb9607-e63d-4ac9-906a-77799dee22d6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:16:54.997328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:54.998321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:54.999190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 245/498] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:16:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:54.999851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdb9607-e63d-4ac9-906a-77799dee22d6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.032237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:55.088724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 254/499] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:55.089905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fd71dc-b79c-4b9d-82f0-47be536e9d2a None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.110364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdb9607-e63d-4ac9-906a-77799dee22d6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.150999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8ed426-e8aa-4235-ae28-8935ceaba72e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.151655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8ed426-e8aa-4235-ae28-8935ceaba72e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.151992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8ed426-e8aa-4235-ae28-8935ceaba72e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.152202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8ed426-e8aa-4235-ae28-8935ceaba72e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.163754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8ed426-e8aa-4235-ae28-8935ceaba72e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.165248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8ed426-e8aa-4235-ae28-8935ceaba72e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-2b8ed426-e8aa-4235-ae28-8935ceaba72e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:55.181133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:55.182050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:55.182908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:55.182908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:55.182908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:55.182908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 246/500] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:55.183841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8ed426-e8aa-4235-ae28-8935ceaba72e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.208102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fd71dc-b79c-4b9d-82f0-47be536e9d2a None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.224155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:55.253237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a18fc1-0b49-4cb7-9ed9-e07f1638495b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.254275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a18fc1-0b49-4cb7-9ed9-e07f1638495b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.255014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a18fc1-0b49-4cb7-9ed9-e07f1638495b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.255426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a18fc1-0b49-4cb7-9ed9-e07f1638495b None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.256215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a18fc1-0b49-4cb7-9ed9-e07f1638495b None admin] RBAC: Adding query filter params (name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:55.256822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a18fc1-0b49-4cb7-9ed9-e07f1638495b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.258935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a18fc1-0b49-4cb7-9ed9-e07f1638495b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:55.273325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 255/501] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:55.273325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a18fc1-0b49-4cb7-9ed9-e07f1638495b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.289893 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:55.302733 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8ed426-e8aa-4235-ae28-8935ceaba72e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.338486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de99814-cdfe-40d3-bf45-4934b15b2dc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.339096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de99814-cdfe-40d3-bf45-4934b15b2dc0 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.339405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de99814-cdfe-40d3-bf45-4934b15b2dc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.339679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de99814-cdfe-40d3-bf45-4934b15b2dc0 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.340348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de99814-cdfe-40d3-bf45-4934b15b2dc0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:55.340649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de99814-cdfe-40d3-bf45-4934b15b2dc0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.342510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de99814-cdfe-40d3-bf45-4934b15b2dc0 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:55.350940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 247/502] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:55.351731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de99814-cdfe-40d3-bf45-4934b15b2dc0 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.389425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a18fc1-0b49-4cb7-9ed9-e07f1638495b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.425382 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a18eec1-828f-4589-ae09-14320bf673e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.425953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a18eec1-828f-4589-ae09-14320bf673e5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.426533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a18eec1-828f-4589-ae09-14320bf673e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.426803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a18eec1-828f-4589-ae09-14320bf673e5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.439091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a18eec1-828f-4589-ae09-14320bf673e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.440594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a18eec1-828f-4589-ae09-14320bf673e5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:55.456893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-4a18eec1-828f-4589-ae09-14320bf673e5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:55.457103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:55.457951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:55.458840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:55.458840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:55.458840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:55.458840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 256/503] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:55.459050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a18eec1-828f-4589-ae09-14320bf673e5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.466968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de99814-cdfe-40d3-bf45-4934b15b2dc0 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.502608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2d2000-f55a-4e23-a142-477fb7df3759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.503227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2d2000-f55a-4e23-a142-477fb7df3759 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.503505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2d2000-f55a-4e23-a142-477fb7df3759 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.503772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2d2000-f55a-4e23-a142-477fb7df3759 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.504407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2d2000-f55a-4e23-a142-477fb7df3759 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:55.504714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2d2000-f55a-4e23-a142-477fb7df3759 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.506084 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2d2000-f55a-4e23-a142-477fb7df3759 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:55.518348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 248/504] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:55.519198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2d2000-f55a-4e23-a142-477fb7df3759 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.544525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:55.587327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a18eec1-828f-4589-ae09-14320bf673e5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.620302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2d2000-f55a-4e23-a142-477fb7df3759 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.625661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873f0311-3c8c-43bb-a29a-e4a9d414ff10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.626330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f0311-3c8c-43bb-a29a-e4a9d414ff10 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.626607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f0311-3c8c-43bb-a29a-e4a9d414ff10 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.626868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873f0311-3c8c-43bb-a29a-e4a9d414ff10 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.638551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873f0311-3c8c-43bb-a29a-e4a9d414ff10 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.640396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873f0311-3c8c-43bb-a29a-e4a9d414ff10 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-873f0311-3c8c-43bb-a29a-e4a9d414ff10 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:55.655249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:55.656220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:55.661844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:55.661844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:55.661844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:55.662005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fe3fda-c0d7-4eb3-bf0f-e91100c429fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.662142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 257/505] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:55.662216 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fe3fda-c0d7-4eb3-bf0f-e91100c429fc None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.662282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873f0311-3c8c-43bb-a29a-e4a9d414ff10 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.662346 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fe3fda-c0d7-4eb3-bf0f-e91100c429fc None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.662346 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fe3fda-c0d7-4eb3-bf0f-e91100c429fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.667675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fe3fda-c0d7-4eb3-bf0f-e91100c429fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.669165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fe3fda-c0d7-4eb3-bf0f-e91100c429fc None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-35fe3fda-c0d7-4eb3-bf0f-e91100c429fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:55.683910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:55.684847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:55.685654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:55.685654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:55.685654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:55.685654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 249/506] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:55.685654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fe3fda-c0d7-4eb3-bf0f-e91100c429fc None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.696516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:55.761609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873f0311-3c8c-43bb-a29a-e4a9d414ff10 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.777252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:55.785377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fe3fda-c0d7-4eb3-bf0f-e91100c429fc None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.799427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec242a6a-77d3-4b39-938d-64fdf3ebd9f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.800306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec242a6a-77d3-4b39-938d-64fdf3ebd9f3 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.800584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec242a6a-77d3-4b39-938d-64fdf3ebd9f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.801056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec242a6a-77d3-4b39-938d-64fdf3ebd9f3 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.801770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec242a6a-77d3-4b39-938d-64fdf3ebd9f3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:55.802093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec242a6a-77d3-4b39-938d-64fdf3ebd9f3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.803494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec242a6a-77d3-4b39-938d-64fdf3ebd9f3 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:55.815683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 258/507] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:55.816889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6fa097-548e-4295-8abd-60a08dc74f45 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.817173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6fa097-548e-4295-8abd-60a08dc74f45 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.817445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6fa097-548e-4295-8abd-60a08dc74f45 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.820569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 259/508] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:55.820569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6fa097-548e-4295-8abd-60a08dc74f45 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.821179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c97eb2-4753-488b-ba55-249348513918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.821811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c97eb2-4753-488b-ba55-249348513918 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.822082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c97eb2-4753-488b-ba55-249348513918 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.822375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c97eb2-4753-488b-ba55-249348513918 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.830128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c97eb2-4753-488b-ba55-249348513918 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.831611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c97eb2-4753-488b-ba55-249348513918 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-e3c97eb2-4753-488b-ba55-249348513918 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:16:55.844188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:55.845089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:55.845952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 250/509] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/users/glance => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:55.846581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c97eb2-4753-488b-ba55-249348513918 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.916515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:55.929658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6fa097-548e-4295-8abd-60a08dc74f45 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.952513 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c97eb2-4753-488b-ba55-249348513918 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:55.964483 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c42860-1ddb-48d0-80cd-76ee0ddf9a5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.965093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c42860-1ddb-48d0-80cd-76ee0ddf9a5b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.965367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c42860-1ddb-48d0-80cd-76ee0ddf9a5b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.965631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c42860-1ddb-48d0-80cd-76ee0ddf9a5b None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.966310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c42860-1ddb-48d0-80cd-76ee0ddf9a5b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:55.966612 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c42860-1ddb-48d0-80cd-76ee0ddf9a5b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.968535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c42860-1ddb-48d0-80cd-76ee0ddf9a5b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:55.980121 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 260/510] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:55.981081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c42860-1ddb-48d0-80cd-76ee0ddf9a5b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:55.989837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c23d4b7-fcc0-41f0-af09-7baf639f73a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:55.990494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c23d4b7-fcc0-41f0-af09-7baf639f73a2 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:55.990767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c23d4b7-fcc0-41f0-af09-7baf639f73a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:55.991050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c23d4b7-fcc0-41f0-af09-7baf639f73a2 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:55.991689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c23d4b7-fcc0-41f0-af09-7baf639f73a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:55.991988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c23d4b7-fcc0-41f0-af09-7baf639f73a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:55.993391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c23d4b7-fcc0-41f0-af09-7baf639f73a2 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.005660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 251/511] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:56.006860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e472101-b3a5-4cdd-b894-4ceeebc38411 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.007145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e472101-b3a5-4cdd-b894-4ceeebc38411 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.007417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e472101-b3a5-4cdd-b894-4ceeebc38411 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.019668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:56.025194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3e472101-b3a5-4cdd-b894-4ceeebc38411 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:56.026437 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3e472101-b3a5-4cdd-b894-4ceeebc38411 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:56.082026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c42860-1ddb-48d0-80cd-76ee0ddf9a5b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:56.090801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 252/512] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:16:56.091636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e472101-b3a5-4cdd-b894-4ceeebc38411 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:56.114907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d80794-972a-450d-acf2-36733cc3fb97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:56.115626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d80794-972a-450d-acf2-36733cc3fb97 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.116016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d80794-972a-450d-acf2-36733cc3fb97 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.116347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d80794-972a-450d-acf2-36733cc3fb97 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.123332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d80794-972a-450d-acf2-36733cc3fb97 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:56.125168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d80794-972a-450d-acf2-36733cc3fb97 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-19d80794-972a-450d-acf2-36733cc3fb97 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:16:56.139234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:56.140476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:56.141572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:16:56.141572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:16:56.141572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:16:56.141572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:16:56.141572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:16:56.141572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:16:56.141572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:16:56.141572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:56.142160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 261/513] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:56.142899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d80794-972a-450d-acf2-36733cc3fb97 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:56.170803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:56.218289 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e472101-b3a5-4cdd-b894-4ceeebc38411 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:56.246549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:56.253906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d80794-972a-450d-acf2-36733cc3fb97 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:56.257869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cf673a-062f-48c2-b752-d2ab128fedc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:56.258588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cf673a-062f-48c2-b752-d2ab128fedc2 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.258933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cf673a-062f-48c2-b752-d2ab128fedc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.259260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cf673a-062f-48c2-b752-d2ab128fedc2 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.260051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cf673a-062f-48c2-b752-d2ab128fedc2 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:56.260414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cf673a-062f-48c2-b752-d2ab128fedc2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:56.261918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cf673a-062f-48c2-b752-d2ab128fedc2 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.290984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 253/514] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:56.291920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cf673a-062f-48c2-b752-d2ab128fedc2 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:56.294049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063ef75e-aa84-4bb0-b7b6-5162e7c974f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:56.294754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063ef75e-aa84-4bb0-b7b6-5162e7c974f8 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.295034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063ef75e-aa84-4bb0-b7b6-5162e7c974f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.295309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063ef75e-aa84-4bb0-b7b6-5162e7c974f8 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.303193 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063ef75e-aa84-4bb0-b7b6-5162e7c974f8 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:56.304775 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063ef75e-aa84-4bb0-b7b6-5162e7c974f8 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-063ef75e-aa84-4bb0-b7b6-5162e7c974f8 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:16:56.317679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:56.318764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 262/515] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/users/neutron => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:56.319662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063ef75e-aa84-4bb0-b7b6-5162e7c974f8 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:56.397132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cf673a-062f-48c2-b752-d2ab128fedc2 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:56.413488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:56.426635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063ef75e-aa84-4bb0-b7b6-5162e7c974f8 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:56.432449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27476162-c9a2-4aef-ba51-8dcbc1fcbbaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:56.433085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27476162-c9a2-4aef-ba51-8dcbc1fcbbaf None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.433362 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27476162-c9a2-4aef-ba51-8dcbc1fcbbaf None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.433652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27476162-c9a2-4aef-ba51-8dcbc1fcbbaf None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.441040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27476162-c9a2-4aef-ba51-8dcbc1fcbbaf None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:56.442574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27476162-c9a2-4aef-ba51-8dcbc1fcbbaf None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-27476162-c9a2-4aef-ba51-8dcbc1fcbbaf None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:16:56.453957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:56.455013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 254/516] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/users/cinder => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:56.455891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27476162-c9a2-4aef-ba51-8dcbc1fcbbaf None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:56.464874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa51ac21-4fed-4ede-ac3f-dff3760c47ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:56.465510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa51ac21-4fed-4ede-ac3f-dff3760c47ce None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.465793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa51ac21-4fed-4ede-ac3f-dff3760c47ce None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.466067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa51ac21-4fed-4ede-ac3f-dff3760c47ce None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.466796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa51ac21-4fed-4ede-ac3f-dff3760c47ce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:56.467104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa51ac21-4fed-4ede-ac3f-dff3760c47ce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:56.468644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa51ac21-4fed-4ede-ac3f-dff3760c47ce None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.486405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 263/517] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:56.487219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa51ac21-4fed-4ede-ac3f-dff3760c47ce None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:56.505274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:56.555834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27476162-c9a2-4aef-ba51-8dcbc1fcbbaf None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:56.586257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa51ac21-4fed-4ede-ac3f-dff3760c47ce None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:56.589535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d48374-3060-4b68-8a69-2ebc636c9235 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:56.590162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d48374-3060-4b68-8a69-2ebc636c9235 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.590456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d48374-3060-4b68-8a69-2ebc636c9235 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.590731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d48374-3060-4b68-8a69-2ebc636c9235 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.624789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dd0e2e-6d1d-4e50-b4b7-e01df347382e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:56.625397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dd0e2e-6d1d-4e50-b4b7-e01df347382e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.625672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dd0e2e-6d1d-4e50-b4b7-e01df347382e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.625941 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dd0e2e-6d1d-4e50-b4b7-e01df347382e None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.629288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dd0e2e-6d1d-4e50-b4b7-e01df347382e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:56.629288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dd0e2e-6d1d-4e50-b4b7-e01df347382e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:56.629288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dd0e2e-6d1d-4e50-b4b7-e01df347382e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.638207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:56.664544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d48374-3060-4b68-8a69-2ebc636c9235 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:56.666040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d48374-3060-4b68-8a69-2ebc636c9235 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.670806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 264/518] 10.222.0.16 () {56 vars in 1139 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:56.670806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dd0e2e-6d1d-4e50-b4b7-e01df347382e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:56.749216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:56.765217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 255/519] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:56.766194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d48374-3060-4b68-8a69-2ebc636c9235 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:56.775767 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dd0e2e-6d1d-4e50-b4b7-e01df347382e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:56.810761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68911fa8-1447-4831-ae59-3ef1b463e335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:56.811517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68911fa8-1447-4831-ae59-3ef1b463e335 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.811780 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68911fa8-1447-4831-ae59-3ef1b463e335 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.812057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68911fa8-1447-4831-ae59-3ef1b463e335 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.822370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68911fa8-1447-4831-ae59-3ef1b463e335 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:56.823922 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68911fa8-1447-4831-ae59-3ef1b463e335 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-68911fa8-1447-4831-ae59-3ef1b463e335 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:56.838220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:56.839258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:56.843715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:56.843715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:56.843715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:56.843715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 265/520] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:56.843715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68911fa8-1447-4831-ae59-3ef1b463e335 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:56.845579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:56.869277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d48374-3060-4b68-8a69-2ebc636c9235 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:56.906122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4546c8-ecb8-46b6-8019-0fb8e2565cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:56.906750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4546c8-ecb8-46b6-8019-0fb8e2565cd4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.907023 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4546c8-ecb8-46b6-8019-0fb8e2565cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.907292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4546c8-ecb8-46b6-8019-0fb8e2565cd4 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.908021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4546c8-ecb8-46b6-8019-0fb8e2565cd4 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:56.908319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4546c8-ecb8-46b6-8019-0fb8e2565cd4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:56.909763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4546c8-ecb8-46b6-8019-0fb8e2565cd4 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.927635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 256/521] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:56.928501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4546c8-ecb8-46b6-8019-0fb8e2565cd4 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:56.942942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68911fa8-1447-4831-ae59-3ef1b463e335 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:56.969404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:56.979266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0efbe1-0497-4771-9820-c12524550db2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:56.979867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0efbe1-0497-4771-9820-c12524550db2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:56.980139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0efbe1-0497-4771-9820-c12524550db2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:56.980406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0efbe1-0497-4771-9820-c12524550db2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:56.981127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0efbe1-0497-4771-9820-c12524550db2 None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=f027e913f53447a988f74ae4dff84b2c) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:56.981433 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0efbe1-0497-4771-9820-c12524550db2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:56.983050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0efbe1-0497-4771-9820-c12524550db2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:56.989179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 266/522] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/role_assignments?user.id=f027e913f53447a988f74ae4dff84b2c&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:56.989964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0efbe1-0497-4771-9820-c12524550db2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:57.038397 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4546c8-ecb8-46b6-8019-0fb8e2565cd4 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:57.081555 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc7c2a8-ed6b-4ab2-85c4-58db6145919f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:57.082214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc7c2a8-ed6b-4ab2-85c4-58db6145919f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:57.082494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc7c2a8-ed6b-4ab2-85c4-58db6145919f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:57.082767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc7c2a8-ed6b-4ab2-85c4-58db6145919f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:57.085765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:57.101312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0efbe1-0497-4771-9820-c12524550db2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:57.138477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0af0fe-c667-4f73-bd7d-ed6f5dee84e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:57.139163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0af0fe-c667-4f73-bd7d-ed6f5dee84e4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:57.139437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0af0fe-c667-4f73-bd7d-ed6f5dee84e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:57.139702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0af0fe-c667-4f73-bd7d-ed6f5dee84e4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:57.152003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0af0fe-c667-4f73-bd7d-ed6f5dee84e4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:57.153488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0af0fe-c667-4f73-bd7d-ed6f5dee84e4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:57.161016 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc7c2a8-ed6b-4ab2-85c4-58db6145919f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:57.162553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc7c2a8-ed6b-4ab2-85c4-58db6145919f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-da0af0fe-c667-4f73-bd7d-ed6f5dee84e4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:57.168519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:57.169431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:57.170345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:57.170345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:57.170345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:57.175137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 267/523] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:57.176054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0af0fe-c667-4f73-bd7d-ed6f5dee84e4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:57.222480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:57.267682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 257/524] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:57.268543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc7c2a8-ed6b-4ab2-85c4-58db6145919f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:57.286540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0af0fe-c667-4f73-bd7d-ed6f5dee84e4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:57.304291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:57.322949 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2323e51e-c2a9-4183-aa76-7678770854b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:57.323572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2323e51e-c2a9-4183-aa76-7678770854b7 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:57.323850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2323e51e-c2a9-4183-aa76-7678770854b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:57.324122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2323e51e-c2a9-4183-aa76-7678770854b7 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:57.325199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2323e51e-c2a9-4183-aa76-7678770854b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:57.325508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2323e51e-c2a9-4183-aa76-7678770854b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:57.327067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2323e51e-c2a9-4183-aa76-7678770854b7 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:57.338865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 268/525] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:57.339686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2323e51e-c2a9-4183-aa76-7678770854b7 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:57.383817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc7c2a8-ed6b-4ab2-85c4-58db6145919f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:57.422234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fd753e-4ccb-4df7-a70c-568754e382c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:57.422858 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fd753e-4ccb-4df7-a70c-568754e382c8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:57.423406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fd753e-4ccb-4df7-a70c-568754e382c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:57.423514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fd753e-4ccb-4df7-a70c-568754e382c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:57.449018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2323e51e-c2a9-4183-aa76-7678770854b7 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:57.450589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:57.491719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d77101-938f-4d30-af7d-ca1cc42aa14d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:57.492321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d77101-938f-4d30-af7d-ca1cc42aa14d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:57.492595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d77101-938f-4d30-af7d-ca1cc42aa14d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:57.493015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d77101-938f-4d30-af7d-ca1cc42aa14d None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:57.493510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d77101-938f-4d30-af7d-ca1cc42aa14d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:57.493819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d77101-938f-4d30-af7d-ca1cc42aa14d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:57.495218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d77101-938f-4d30-af7d-ca1cc42aa14d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:57.500093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fd753e-4ccb-4df7-a70c-568754e382c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:57.501576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fd753e-4ccb-4df7-a70c-568754e382c8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:57.507761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 269/526] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:57.508412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d77101-938f-4d30-af7d-ca1cc42aa14d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:57.542949 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:57.608093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d77101-938f-4d30-af7d-ca1cc42aa14d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:57.618426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 258/527] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:16:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:16:57.618426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fd753e-4ccb-4df7-a70c-568754e382c8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:57.652587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141524bb-f38d-4b56-9e6b-963e33ecbe1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:57.653238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141524bb-f38d-4b56-9e6b-963e33ecbe1e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:57.653516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141524bb-f38d-4b56-9e6b-963e33ecbe1e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:57.653788 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141524bb-f38d-4b56-9e6b-963e33ecbe1e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:57.665488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141524bb-f38d-4b56-9e6b-963e33ecbe1e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:57.667113 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141524bb-f38d-4b56-9e6b-963e33ecbe1e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-141524bb-f38d-4b56-9e6b-963e33ecbe1e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:57.681771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:57.682979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:57.683843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:57.683843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:57.683843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:57.683843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 270/528] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:57.684057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141524bb-f38d-4b56-9e6b-963e33ecbe1e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:57.701339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:57.733508 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fd753e-4ccb-4df7-a70c-568754e382c8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:57.769353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6544da5-adaa-4e6d-a413-c3de0a7f3594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:57.769985 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6544da5-adaa-4e6d-a413-c3de0a7f3594 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:57.770287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6544da5-adaa-4e6d-a413-c3de0a7f3594 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:57.770557 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6544da5-adaa-4e6d-a413-c3de0a7f3594 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:57.771323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6544da5-adaa-4e6d-a413-c3de0a7f3594 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:57.772871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6544da5-adaa-4e6d-a413-c3de0a7f3594 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:57.781616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:57.793005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141524bb-f38d-4b56-9e6b-963e33ecbe1e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:57.806666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 259/529] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:16:57 2022] POST /identity/v3/users => generated 247 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:16:57.807697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6544da5-adaa-4e6d-a413-c3de0a7f3594 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:57.835218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d95f00-fe1b-412e-8749-ffab35c03a1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:57.836768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d95f00-fe1b-412e-8749-ffab35c03a1e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:57.837588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d95f00-fe1b-412e-8749-ffab35c03a1e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:57.838185 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d95f00-fe1b-412e-8749-ffab35c03a1e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:57.847315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d95f00-fe1b-412e-8749-ffab35c03a1e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:57.849213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d95f00-fe1b-412e-8749-ffab35c03a1e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-27d95f00-fe1b-412e-8749-ffab35c03a1e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:16:57.863558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:57.864792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:57.865605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:16:57.865605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:16:57.865605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:16:57.865605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:16:57.865605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:16:57.865605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:16:57.865605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:16:57.865605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:16:57.867862 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 271/530] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:57.869068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d95f00-fe1b-412e-8749-ffab35c03a1e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:57.927170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6544da5-adaa-4e6d-a413-c3de0a7f3594 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:57.963775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2c4b8d-9e7e-4c74-a567-dade6d852e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:57.964444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2c4b8d-9e7e-4c74-a567-dade6d852e60 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:57.964729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2c4b8d-9e7e-4c74-a567-dade6d852e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:57.964995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2c4b8d-9e7e-4c74-a567-dade6d852e60 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:57.973443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:57.977282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2c4b8d-9e7e-4c74-a567-dade6d852e60 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:57.979008 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2c4b8d-9e7e-4c74-a567-dade6d852e60 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-ab2c4b8d-9e7e-4c74-a567-dade6d852e60 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:16:57.995779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:57.996762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:16:57.997593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:16:57.997593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:16:57.997593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:57.997593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 260/531] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:16:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:57.997834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2c4b8d-9e7e-4c74-a567-dade6d852e60 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:58.012914 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d95f00-fe1b-412e-8749-ffab35c03a1e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:58.057240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:58.067028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634d2e16-82a0-4273-93c2-c9d9f9a88ebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:58.068763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634d2e16-82a0-4273-93c2-c9d9f9a88ebd None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:58.068763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634d2e16-82a0-4273-93c2-c9d9f9a88ebd None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:58.068763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634d2e16-82a0-4273-93c2-c9d9f9a88ebd None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:58.068978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634d2e16-82a0-4273-93c2-c9d9f9a88ebd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:58.071219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634d2e16-82a0-4273-93c2-c9d9f9a88ebd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:58.071219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634d2e16-82a0-4273-93c2-c9d9f9a88ebd None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:58.084827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 272/532] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:58.086055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634d2e16-82a0-4273-93c2-c9d9f9a88ebd None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:58.121117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2c4b8d-9e7e-4c74-a567-dade6d852e60 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:58.159277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88b6418-7ba5-4b54-8954-710d37fd4eae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:58.159923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b6418-7ba5-4b54-8954-710d37fd4eae None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:58.160204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b6418-7ba5-4b54-8954-710d37fd4eae None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:58.160532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b6418-7ba5-4b54-8954-710d37fd4eae None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:58.161232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88b6418-7ba5-4b54-8954-710d37fd4eae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:58.161584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88b6418-7ba5-4b54-8954-710d37fd4eae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:58.163577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88b6418-7ba5-4b54-8954-710d37fd4eae None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:58.186184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 261/533] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:16:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:58.187140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88b6418-7ba5-4b54-8954-710d37fd4eae None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:58.191229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634d2e16-82a0-4273-93c2-c9d9f9a88ebd None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:58.233824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfe6d0d-195e-4a5b-931f-cfdd8c51ea6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:58.234528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfe6d0d-195e-4a5b-931f-cfdd8c51ea6e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:58.234807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfe6d0d-195e-4a5b-931f-cfdd8c51ea6e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:58.235075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfe6d0d-195e-4a5b-931f-cfdd8c51ea6e None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:58.235836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfe6d0d-195e-4a5b-931f-cfdd8c51ea6e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:58.236144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfe6d0d-195e-4a5b-931f-cfdd8c51ea6e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:58.238120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfe6d0d-195e-4a5b-931f-cfdd8c51ea6e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:58.240480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:58.256087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 273/534] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:16:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:58.257231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfe6d0d-195e-4a5b-931f-cfdd8c51ea6e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:58.303249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88b6418-7ba5-4b54-8954-710d37fd4eae None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:58.304910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:58.340532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0a8e81-6b84-4afa-99d6-d454acb2eed1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:58.341270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0a8e81-6b84-4afa-99d6-d454acb2eed1 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:58.341630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0a8e81-6b84-4afa-99d6-d454acb2eed1 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:58.341960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0a8e81-6b84-4afa-99d6-d454acb2eed1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:58.349605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0a8e81-6b84-4afa-99d6-d454acb2eed1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:58.351312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0a8e81-6b84-4afa-99d6-d454acb2eed1 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-8c0a8e81-6b84-4afa-99d6-d454acb2eed1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:16:58.364267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:58.365125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:58.366172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:16:58.366172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:16:58.366172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:16:58.366172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:16:58.366172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:16:58.366172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:16:58.366172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:16:58.366172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:58.366172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 262/535] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:58.367733 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0a8e81-6b84-4afa-99d6-d454acb2eed1 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:58.391444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfe6d0d-195e-4a5b-931f-cfdd8c51ea6e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:58.436109 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32bbc9f-8934-45a1-8491-db99ea15ab7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:58.436820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32bbc9f-8934-45a1-8491-db99ea15ab7e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:58.437105 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32bbc9f-8934-45a1-8491-db99ea15ab7e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:58.437377 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32bbc9f-8934-45a1-8491-db99ea15ab7e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:58.438271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32bbc9f-8934-45a1-8491-db99ea15ab7e None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=2ce646dcb8124685ac7a1c1eca664d04) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:58.438574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32bbc9f-8934-45a1-8491-db99ea15ab7e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:58.440261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32bbc9f-8934-45a1-8491-db99ea15ab7e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:58.450278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 274/536] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:58 2022] GET /identity/v3/role_assignments?user.id=2ce646dcb8124685ac7a1c1eca664d04&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:58.452306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32bbc9f-8934-45a1-8491-db99ea15ab7e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:58.512587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0a8e81-6b84-4afa-99d6-d454acb2eed1 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:58.572104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed8b998-7ab8-4dcc-b809-10dd65faf779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:58.572943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed8b998-7ab8-4dcc-b809-10dd65faf779 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:58.573312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed8b998-7ab8-4dcc-b809-10dd65faf779 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:58.573670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed8b998-7ab8-4dcc-b809-10dd65faf779 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:58.576328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:58.582950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed8b998-7ab8-4dcc-b809-10dd65faf779 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:58.584831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed8b998-7ab8-4dcc-b809-10dd65faf779 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-eed8b998-7ab8-4dcc-b809-10dd65faf779 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:16:58.599292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:16:58.600202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:16:58.601051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:16:58.601051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:16:58.601051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:16:58.601051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:16:58.601051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:16:58.601051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:16:58.601051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:16:58.601051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:16:58.603288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 263/537] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:16:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:16:58.608952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed8b998-7ab8-4dcc-b809-10dd65faf779 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:58.673402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:58.687749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32bbc9f-8934-45a1-8491-db99ea15ab7e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:58.731573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e8cdab-bfde-45d2-955b-db997f41c397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:58.732197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e8cdab-bfde-45d2-955b-db997f41c397 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:58.732472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e8cdab-bfde-45d2-955b-db997f41c397 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:58.733107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e8cdab-bfde-45d2-955b-db997f41c397 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:58.733815 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e8cdab-bfde-45d2-955b-db997f41c397 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:58.734196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e8cdab-bfde-45d2-955b-db997f41c397 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:58.739122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e8cdab-bfde-45d2-955b-db997f41c397 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:58.766532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 275/538] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:16:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:58.772190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e8cdab-bfde-45d2-955b-db997f41c397 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:58.826714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed8b998-7ab8-4dcc-b809-10dd65faf779 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:58.903178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e8cdab-bfde-45d2-955b-db997f41c397 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:58.910893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0014e23-3e2e-4bf3-9bae-7731ac0a690e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:58.911589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0014e23-3e2e-4bf3-9bae-7731ac0a690e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:58.911882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0014e23-3e2e-4bf3-9bae-7731ac0a690e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:58.912161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0014e23-3e2e-4bf3-9bae-7731ac0a690e None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:58.912873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0014e23-3e2e-4bf3-9bae-7731ac0a690e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:58.913181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0014e23-3e2e-4bf3-9bae-7731ac0a690e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:58.914687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0014e23-3e2e-4bf3-9bae-7731ac0a690e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:58.934505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:58.937468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 264/539] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:16:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:58.939685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68da3b52-6bc9-4f05-a4a0-4302c15addc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:58.940462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68da3b52-6bc9-4f05-a4a0-4302c15addc4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:58.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68da3b52-6bc9-4f05-a4a0-4302c15addc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:58.941132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68da3b52-6bc9-4f05-a4a0-4302c15addc4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:58.941929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68da3b52-6bc9-4f05-a4a0-4302c15addc4 None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=171e916244804add8f18b44b77b03d72) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:58.942326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68da3b52-6bc9-4f05-a4a0-4302c15addc4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:58.943823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68da3b52-6bc9-4f05-a4a0-4302c15addc4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:58.946604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0014e23-3e2e-4bf3-9bae-7731ac0a690e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:16:58.950043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 276/540] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:58 2022] GET /identity/v3/role_assignments?user.id=171e916244804add8f18b44b77b03d72&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:16:59.061757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:16:59.088604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0014e23-3e2e-4bf3-9bae-7731ac0a690e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:16:59.139790 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fe5457-2e74-4f82-9e54-ef1ea58a211e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:16:59.140420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe5457-2e74-4f82-9e54-ef1ea58a211e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:59.140694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe5457-2e74-4f82-9e54-ef1ea58a211e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:59.140970 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe5457-2e74-4f82-9e54-ef1ea58a211e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:59.141711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fe5457-2e74-4f82-9e54-ef1ea58a211e None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=39c54901c5b5400f96e084fd4adf54e1) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:16:59.142018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fe5457-2e74-4f82-9e54-ef1ea58a211e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:16:59.143492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fe5457-2e74-4f82-9e54-ef1ea58a211e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:16:59.154692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 265/541] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:16:58 2022] GET /identity/v3/role_assignments?user.id=39c54901c5b5400f96e084fd4adf54e1&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 259 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:16:59.513496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f4f1a8-0df6-4013-8f48-93d5ea10c1d3 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:59.514423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f4f1a8-0df6-4013-8f48-93d5ea10c1d3 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:59.514423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f4f1a8-0df6-4013-8f48-93d5ea10c1d3 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:59.515898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 277/542] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:16:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:16:59.521864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2061c90c-82e1-416c-b2c7-e5def4c7fc3f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:16:59.522198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2061c90c-82e1-416c-b2c7-e5def4c7fc3f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:16:59.522434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2061c90c-82e1-416c-b2c7-e5def4c7fc3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:16:59.545955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2061c90c-82e1-416c-b2c7-e5def4c7fc3f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:16:59.547193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2061c90c-82e1-416c-b2c7-e5def4c7fc3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:16:59.652051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 266/543] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:16:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:00.040712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f4f1a8-0df6-4013-8f48-93d5ea10c1d3 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:00.159312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f4f1a8-0df6-4013-8f48-93d5ea10c1d3 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:00.203872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57147e37-f121-45ec-9c28-17fe4e422bab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:00.204512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57147e37-f121-45ec-9c28-17fe4e422bab None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:00.204789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57147e37-f121-45ec-9c28-17fe4e422bab None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:00.205060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57147e37-f121-45ec-9c28-17fe4e422bab None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:00.227076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57147e37-f121-45ec-9c28-17fe4e422bab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-57147e37-f121-45ec-9c28-17fe4e422bab None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:00.231980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:00.233909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:00.235004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 278/544] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:00.239530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2061c90c-82e1-416c-b2c7-e5def4c7fc3f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:00.363266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:00.395949 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2061c90c-82e1-416c-b2c7-e5def4c7fc3f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:00.444760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196a0d94-06ef-49fe-855b-72f9fe372faa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:00.445390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196a0d94-06ef-49fe-855b-72f9fe372faa None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:00.445672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196a0d94-06ef-49fe-855b-72f9fe372faa None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:00.445950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196a0d94-06ef-49fe-855b-72f9fe372faa None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:00.446654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196a0d94-06ef-49fe-855b-72f9fe372faa None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:00.446960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196a0d94-06ef-49fe-855b-72f9fe372faa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:00.448348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196a0d94-06ef-49fe-855b-72f9fe372faa None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:00.457263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 267/545] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:00.462565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57147e37-f121-45ec-9c28-17fe4e422bab None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:00.508000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:00.578008 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57147e37-f121-45ec-9c28-17fe4e422bab None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:00.621777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b66182-e288-444c-bf05-17fdc826b66b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:00.622427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b66182-e288-444c-bf05-17fdc826b66b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:00.622705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b66182-e288-444c-bf05-17fdc826b66b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:00.622978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b66182-e288-444c-bf05-17fdc826b66b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:00.644516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b66182-e288-444c-bf05-17fdc826b66b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:00.646015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b66182-e288-444c-bf05-17fdc826b66b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:00.662151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82797e-5460-4c8c-aeef-d33419f56866 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:00.662494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82797e-5460-4c8c-aeef-d33419f56866 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:00.662777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82797e-5460-4c8c-aeef-d33419f56866 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:00.664374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 268/546] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-c9b66182-e288-444c-bf05-17fdc826b66b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:00.665572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:00.667707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:00.669710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:00.669710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:00.669710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:00.669710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 279/547] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:00.674145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d72f270-e8ce-4ddb-bc9b-56d3ab6133b4 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:00.674145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d72f270-e8ce-4ddb-bc9b-56d3ab6133b4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:00.674145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d72f270-e8ce-4ddb-bc9b-56d3ab6133b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:00.678772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b66182-e288-444c-bf05-17fdc826b66b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:00.699580 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-6d72f270-e8ce-4ddb-bc9b-56d3ab6133b4 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:00.702406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6d72f270-e8ce-4ddb-bc9b-56d3ab6133b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:00.806453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 269/548] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:00.857840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c4bc92-702e-467a-8f29-358ba4ab146f None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:00.858204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c4bc92-702e-467a-8f29-358ba4ab146f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:00.858459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c4bc92-702e-467a-8f29-358ba4ab146f None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:00.860080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 270/549] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:00.866139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f01a68-e20a-4f19-bd08-deb0bf0a61db None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:00.866139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f01a68-e20a-4f19-bd08-deb0bf0a61db None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:00.866139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f01a68-e20a-4f19-bd08-deb0bf0a61db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:00.874598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:00.888001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b66182-e288-444c-bf05-17fdc826b66b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:00.900278 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-93f01a68-e20a-4f19-bd08-deb0bf0a61db None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:00.903212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-93f01a68-e20a-4f19-bd08-deb0bf0a61db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:00.949115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b82e53-e096-4c65-8150-bc693bc23926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:00.949979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b82e53-e096-4c65-8150-bc693bc23926 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:00.950407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b82e53-e096-4c65-8150-bc693bc23926 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:00.950826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b82e53-e096-4c65-8150-bc693bc23926 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:00.951692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b82e53-e096-4c65-8150-bc693bc23926 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:00.953176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b82e53-e096-4c65-8150-bc693bc23926 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:00.954716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b82e53-e096-4c65-8150-bc693bc23926 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:00.974989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 280/550] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:00.981081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b82e53-e096-4c65-8150-bc693bc23926 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:01.026568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:01.038327 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 271/551] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:17:01.113347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f01a68-e20a-4f19-bd08-deb0bf0a61db None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:01.179946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b82e53-e096-4c65-8150-bc693bc23926 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:01.241067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08d28ad-d898-4b28-922e-499ca42b4b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:01.241714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08d28ad-d898-4b28-922e-499ca42b4b22 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.241992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08d28ad-d898-4b28-922e-499ca42b4b22 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.242294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08d28ad-d898-4b28-922e-499ca42b4b22 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.256759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08d28ad-d898-4b28-922e-499ca42b4b22 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:01.263035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08d28ad-d898-4b28-922e-499ca42b4b22 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-e08d28ad-d898-4b28-922e-499ca42b4b22 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:01.283194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:01.284395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:01.285908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 281/552] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Sep 13 15:17:00 2022] GET /identity/v3/users/nova => generated 82 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:01.290604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08d28ad-d898-4b28-922e-499ca42b4b22 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:01.306376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f01a68-e20a-4f19-bd08-deb0bf0a61db None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:01.327658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:01.350379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a5a50f-dae7-4ca6-a6dc-4bc46c53737a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:01.351488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a5a50f-dae7-4ca6-a6dc-4bc46c53737a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.351954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a5a50f-dae7-4ca6-a6dc-4bc46c53737a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.352439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a5a50f-dae7-4ca6-a6dc-4bc46c53737a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.366219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a5a50f-dae7-4ca6-a6dc-4bc46c53737a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-16a5a50f-dae7-4ca6-a6dc-4bc46c53737a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:01.373072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:01.374267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:01.375082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:01.378649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 272/553] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:01.379757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a5a50f-dae7-4ca6-a6dc-4bc46c53737a None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:01.442048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08d28ad-d898-4b28-922e-499ca42b4b22 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:01.451043 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:01.478963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a3c557-e4e7-45ad-a57c-b8b3b2131e80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:01.479900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a3c557-e4e7-45ad-a57c-b8b3b2131e80 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.480351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a3c557-e4e7-45ad-a57c-b8b3b2131e80 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.480790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a3c557-e4e7-45ad-a57c-b8b3b2131e80 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.481740 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a3c557-e4e7-45ad-a57c-b8b3b2131e80 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:01.482282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a3c557-e4e7-45ad-a57c-b8b3b2131e80 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:01.484135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a3c557-e4e7-45ad-a57c-b8b3b2131e80 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:01.507186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a5a50f-dae7-4ca6-a6dc-4bc46c53737a None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:01.507588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 282/554] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Sep 13 15:17:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:01.508698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a3c557-e4e7-45ad-a57c-b8b3b2131e80 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:01.548531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b339fa-7eeb-40cb-943f-6e117c62e8a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:01.549172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b339fa-7eeb-40cb-943f-6e117c62e8a5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.549451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b339fa-7eeb-40cb-943f-6e117c62e8a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.549717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b339fa-7eeb-40cb-943f-6e117c62e8a5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.561280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b339fa-7eeb-40cb-943f-6e117c62e8a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-a5b339fa-7eeb-40cb-943f-6e117c62e8a5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:01.566160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:01.567162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 273/555] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:01.568012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b339fa-7eeb-40cb-943f-6e117c62e8a5 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:01.603512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:01.644220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a3c557-e4e7-45ad-a57c-b8b3b2131e80 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:01.680984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625f25d7-417e-403f-81fc-21a37ffb5880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:01.681965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625f25d7-417e-403f-81fc-21a37ffb5880 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.682336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625f25d7-417e-403f-81fc-21a37ffb5880 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.682670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625f25d7-417e-403f-81fc-21a37ffb5880 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.683455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625f25d7-417e-403f-81fc-21a37ffb5880 None admin] RBAC: Adding query filter params (name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:01.683852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625f25d7-417e-403f-81fc-21a37ffb5880 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:01.685290 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625f25d7-417e-403f-81fc-21a37ffb5880 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:01.686094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b339fa-7eeb-40cb-943f-6e117c62e8a5 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:01.697967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 283/556] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:01.697967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62414bb-9dd0-4a2a-aa0e-9a95c04eff85 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.697967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62414bb-9dd0-4a2a-aa0e-9a95c04eff85 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.697967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62414bb-9dd0-4a2a-aa0e-9a95c04eff85 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.697967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 284/557] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:01.697967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62414bb-9dd0-4a2a-aa0e-9a95c04eff85 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:01.710958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:01.731299 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020e8518-e09f-4cbf-8d54-62c582dbe2e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:01.731950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020e8518-e09f-4cbf-8d54-62c582dbe2e7 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.732225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020e8518-e09f-4cbf-8d54-62c582dbe2e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.732498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020e8518-e09f-4cbf-8d54-62c582dbe2e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.744373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020e8518-e09f-4cbf-8d54-62c582dbe2e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:01.745898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020e8518-e09f-4cbf-8d54-62c582dbe2e7 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-020e8518-e09f-4cbf-8d54-62c582dbe2e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:01.761477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:01.763019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:01.763954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:01.763954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:01.763954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:01.763954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 274/558] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:01.763954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020e8518-e09f-4cbf-8d54-62c582dbe2e7 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:01.818273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62414bb-9dd0-4a2a-aa0e-9a95c04eff85 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:01.855577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6df9a69-a6b4-4a02-9a91-835d3a072e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:01.856107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6df9a69-a6b4-4a02-9a91-835d3a072e39 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.856381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6df9a69-a6b4-4a02-9a91-835d3a072e39 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.856651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6df9a69-a6b4-4a02-9a91-835d3a072e39 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.857322 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6df9a69-a6b4-4a02-9a91-835d3a072e39 None admin] RBAC: Adding query filter params (name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:01.857746 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6df9a69-a6b4-4a02-9a91-835d3a072e39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:01.859605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6df9a69-a6b4-4a02-9a91-835d3a072e39 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:01.861026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:01.866813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020e8518-e09f-4cbf-8d54-62c582dbe2e7 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:01.868831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 285/559] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:01.870011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6f61e1-91d0-44c2-a698-72677fc5ea7d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.870335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6f61e1-91d0-44c2-a698-72677fc5ea7d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.870604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6f61e1-91d0-44c2-a698-72677fc5ea7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.889371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-ef6f61e1-91d0-44c2-a698-72677fc5ea7d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:01.890676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ef6f61e1-91d0-44c2-a698-72677fc5ea7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:01.901014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4fdf3e-8117-4c44-8bc2-1b42d6dd3ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:01.901670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4fdf3e-8117-4c44-8bc2-1b42d6dd3ded None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.901951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4fdf3e-8117-4c44-8bc2-1b42d6dd3ded None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.902253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4fdf3e-8117-4c44-8bc2-1b42d6dd3ded None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.913599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4fdf3e-8117-4c44-8bc2-1b42d6dd3ded None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:01.915121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4fdf3e-8117-4c44-8bc2-1b42d6dd3ded None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-9d4fdf3e-8117-4c44-8bc2-1b42d6dd3ded None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:01.929055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:01.929973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:01.930906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:01.930906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:01.930906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:01.930906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 275/560] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:01.931127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4fdf3e-8117-4c44-8bc2-1b42d6dd3ded None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:01.956832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:01.962587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 286/561] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:01.963842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe97d7-0eff-4a43-b934-12bd430267c1 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:01.964305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe97d7-0eff-4a43-b934-12bd430267c1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:01.964412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebe97d7-0eff-4a43-b934-12bd430267c1 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:01.965726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 287/562] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:01 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:01.966298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebe97d7-0eff-4a43-b934-12bd430267c1 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.052198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4fdf3e-8117-4c44-8bc2-1b42d6dd3ded None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.080771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebe97d7-0eff-4a43-b934-12bd430267c1 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.088041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdc5d41-d400-4026-9ade-fd58dc18cc1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.088665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdc5d41-d400-4026-9ade-fd58dc18cc1b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.088941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdc5d41-d400-4026-9ade-fd58dc18cc1b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.089214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdc5d41-d400-4026-9ade-fd58dc18cc1b None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.089855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdc5d41-d400-4026-9ade-fd58dc18cc1b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:02.090180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdc5d41-d400-4026-9ade-fd58dc18cc1b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:02.091584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdc5d41-d400-4026-9ade-fd58dc18cc1b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:02.103304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 276/563] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:02.104206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdc5d41-d400-4026-9ade-fd58dc18cc1b None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.113090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:02.123476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfd98a4-b88d-4264-8c0f-d93d8b043c14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.124363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd98a4-b88d-4264-8c0f-d93d8b043c14 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.124653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd98a4-b88d-4264-8c0f-d93d8b043c14 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.124902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd98a4-b88d-4264-8c0f-d93d8b043c14 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.135171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfd98a4-b88d-4264-8c0f-d93d8b043c14 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:02.136726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfd98a4-b88d-4264-8c0f-d93d8b043c14 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-4dfd98a4-b88d-4264-8c0f-d93d8b043c14 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:02.151488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:02.152444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:02.153315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:02.153315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:02.153315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:02.153315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 288/564] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:02.154334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f30e9b6-8208-4457-9120-013e0413b63a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.154334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f30e9b6-8208-4457-9120-013e0413b63a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.154334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f30e9b6-8208-4457-9120-013e0413b63a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.172344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-1f30e9b6-8208-4457-9120-013e0413b63a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:02.173519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1f30e9b6-8208-4457-9120-013e0413b63a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:02.196903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:02.215884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdc5d41-d400-4026-9ade-fd58dc18cc1b None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.242073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 289/565] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:02.242942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f30e9b6-8208-4457-9120-013e0413b63a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.249667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f26204a-ca92-44ac-be8a-5b00c07afe2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.250311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f26204a-ca92-44ac-be8a-5b00c07afe2f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.250587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f26204a-ca92-44ac-be8a-5b00c07afe2f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.250857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f26204a-ca92-44ac-be8a-5b00c07afe2f None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.251503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f26204a-ca92-44ac-be8a-5b00c07afe2f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:02.251802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f26204a-ca92-44ac-be8a-5b00c07afe2f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:02.253173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f26204a-ca92-44ac-be8a-5b00c07afe2f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:02.269087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 277/566] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:02.269087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f26204a-ca92-44ac-be8a-5b00c07afe2f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.343781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f30e9b6-8208-4457-9120-013e0413b63a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.355287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:02.367386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f26204a-ca92-44ac-be8a-5b00c07afe2f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.381518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad0f8f7-cf3e-44e5-aeff-4309b3b130ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.382243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad0f8f7-cf3e-44e5-aeff-4309b3b130ad None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.382589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad0f8f7-cf3e-44e5-aeff-4309b3b130ad None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.382774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad0f8f7-cf3e-44e5-aeff-4309b3b130ad None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.389328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad0f8f7-cf3e-44e5-aeff-4309b3b130ad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:02.390796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad0f8f7-cf3e-44e5-aeff-4309b3b130ad None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:02.401703 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72da093-031e-4bb3-83b6-ad511c50cbfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-0ad0f8f7-cf3e-44e5-aeff-4309b3b130ad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:02.401932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:02.402848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:02.403720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:02.403720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:02.403720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:02.403720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:02.403720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:02.403720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:02.403720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:02.403720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:02.404163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72da093-031e-4bb3-83b6-ad511c50cbfb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.404163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72da093-031e-4bb3-83b6-ad511c50cbfb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.404163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72da093-031e-4bb3-83b6-ad511c50cbfb None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.404163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72da093-031e-4bb3-83b6-ad511c50cbfb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:02.404163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72da093-031e-4bb3-83b6-ad511c50cbfb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:02.405162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72da093-031e-4bb3-83b6-ad511c50cbfb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:02.408032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 290/567] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:02.408892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad0f8f7-cf3e-44e5-aeff-4309b3b130ad None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.421899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 278/568] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:02.421899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72da093-031e-4bb3-83b6-ad511c50cbfb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.421899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:02.520009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad0f8f7-cf3e-44e5-aeff-4309b3b130ad None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.520517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72da093-031e-4bb3-83b6-ad511c50cbfb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.553071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a773e00-16ce-4637-9116-d47af10ec013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.553503 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0aee05-45e5-4b5d-8ab9-5516db32356f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.553701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a773e00-16ce-4637-9116-d47af10ec013 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.553973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a773e00-16ce-4637-9116-d47af10ec013 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.554163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0aee05-45e5-4b5d-8ab9-5516db32356f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.554279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a773e00-16ce-4637-9116-d47af10ec013 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.554440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0aee05-45e5-4b5d-8ab9-5516db32356f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.554767 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0aee05-45e5-4b5d-8ab9-5516db32356f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.561693 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a773e00-16ce-4637-9116-d47af10ec013 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:02.563281 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a773e00-16ce-4637-9116-d47af10ec013 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:02.565168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0aee05-45e5-4b5d-8ab9-5516db32356f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-3e0aee05-45e5-4b5d-8ab9-5516db32356f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:02.570331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:02.571258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:02.572120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 291/569] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:02.572806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0aee05-45e5-4b5d-8ab9-5516db32356f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-2a773e00-16ce-4637-9116-d47af10ec013 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:02.574221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:02.575322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 279/570] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:02 2022] GET /identity/v3/users/swift => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:02.576322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a773e00-16ce-4637-9116-d47af10ec013 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.605087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:02.641823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:02.677089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0aee05-45e5-4b5d-8ab9-5516db32356f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.681397 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a773e00-16ce-4637-9116-d47af10ec013 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.710166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a37ca40-1eda-4a2e-a2f0-05ba34779790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.710777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a37ca40-1eda-4a2e-a2f0-05ba34779790 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.711048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a37ca40-1eda-4a2e-a2f0-05ba34779790 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.711321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a37ca40-1eda-4a2e-a2f0-05ba34779790 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.712001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a37ca40-1eda-4a2e-a2f0-05ba34779790 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:02.712304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a37ca40-1eda-4a2e-a2f0-05ba34779790 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:02.713730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a37ca40-1eda-4a2e-a2f0-05ba34779790 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:02.715188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d73b42-75e5-4098-8de3-4f8490762d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.715751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d73b42-75e5-4098-8de3-4f8490762d36 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.716047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d73b42-75e5-4098-8de3-4f8490762d36 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.716308 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d73b42-75e5-4098-8de3-4f8490762d36 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.724470 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d73b42-75e5-4098-8de3-4f8490762d36 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:02.725607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 292/571] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:02.725943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d73b42-75e5-4098-8de3-4f8490762d36 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:02.726401 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a37ca40-1eda-4a2e-a2f0-05ba34779790 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-82d73b42-75e5-4098-8de3-4f8490762d36 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:02.740642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:02.741598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 280/572] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:02 2022] GET /identity/v3/users/glance => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:02.742962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d73b42-75e5-4098-8de3-4f8490762d36 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.816804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:02.829427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a37ca40-1eda-4a2e-a2f0-05ba34779790 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.840478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d73b42-75e5-4098-8de3-4f8490762d36 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.862655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:02.864923 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1653d890-27ff-4978-9e4d-9c3d86911906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.865592 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1653d890-27ff-4978-9e4d-9c3d86911906 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.865866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1653d890-27ff-4978-9e4d-9c3d86911906 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.866160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1653d890-27ff-4978-9e4d-9c3d86911906 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.875535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f96ad9a-4b4b-46dc-9a82-c5e6aa62d9a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:02.876147 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f96ad9a-4b4b-46dc-9a82-c5e6aa62d9a2 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:02.876426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f96ad9a-4b4b-46dc-9a82-c5e6aa62d9a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:02.876702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f96ad9a-4b4b-46dc-9a82-c5e6aa62d9a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:02.877373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f96ad9a-4b4b-46dc-9a82-c5e6aa62d9a2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:02.877472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1653d890-27ff-4978-9e4d-9c3d86911906 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:02.877675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f96ad9a-4b4b-46dc-9a82-c5e6aa62d9a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:02.879104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f96ad9a-4b4b-46dc-9a82-c5e6aa62d9a2 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-1653d890-27ff-4978-9e4d-9c3d86911906 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:02.882243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:02.883137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:02.884005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 293/573] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:02.884708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1653d890-27ff-4978-9e4d-9c3d86911906 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.889965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 281/574] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:02.890544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f96ad9a-4b4b-46dc-9a82-c5e6aa62d9a2 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:02.984031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1653d890-27ff-4978-9e4d-9c3d86911906 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:02.988509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f96ad9a-4b4b-46dc-9a82-c5e6aa62d9a2 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:03.018671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bb1049-d760-4ec3-86a7-a435f55b94ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.019278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bb1049-d760-4ec3-86a7-a435f55b94ec None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.019555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bb1049-d760-4ec3-86a7-a435f55b94ec None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.019823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bb1049-d760-4ec3-86a7-a435f55b94ec None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.020542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bb1049-d760-4ec3-86a7-a435f55b94ec None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:03.020846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bb1049-d760-4ec3-86a7-a435f55b94ec None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.022316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bb1049-d760-4ec3-86a7-a435f55b94ec None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.022621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ea3f6a-fe1d-4dec-920a-072f0f15ca99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.023275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ea3f6a-fe1d-4dec-920a-072f0f15ca99 None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.023557 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ea3f6a-fe1d-4dec-920a-072f0f15ca99 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.023826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ea3f6a-fe1d-4dec-920a-072f0f15ca99 None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/f027e913f53447a988f74ae4dff84b2c/roles/da0ea75123ba496fb410e87b8df992ba` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.037997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ea3f6a-fe1d-4dec-920a-072f0f15ca99 None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=f027e913f53447a988f74ae4dff84b2c, role_id=da0ea75123ba496fb410e87b8df992ba)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.039461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ea3f6a-fe1d-4dec-920a-072f0f15ca99 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.039881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 294/575] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Sep 13 15:17:02 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:03.040687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bb1049-d760-4ec3-86a7-a435f55b94ec None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:03.050945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:03.056916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 282/576] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:17:02 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/f027e913f53447a988f74ae4dff84b2c/roles/da0ea75123ba496fb410e87b8df992ba => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:17:03.057739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ea3f6a-fe1d-4dec-920a-072f0f15ca99 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:03.102354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:03.158777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bb1049-d760-4ec3-86a7-a435f55b94ec None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:03.181160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ea3f6a-fe1d-4dec-920a-072f0f15ca99 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:03.196447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12da61e-3fb9-4111-addf-3c2aab5c0e8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.197078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12da61e-3fb9-4111-addf-3c2aab5c0e8d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.197354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12da61e-3fb9-4111-addf-3c2aab5c0e8d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.197625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12da61e-3fb9-4111-addf-3c2aab5c0e8d None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.198386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12da61e-3fb9-4111-addf-3c2aab5c0e8d None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:03.198698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12da61e-3fb9-4111-addf-3c2aab5c0e8d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.200141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12da61e-3fb9-4111-addf-3c2aab5c0e8d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.223824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 295/577] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:03.223824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12da61e-3fb9-4111-addf-3c2aab5c0e8d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:03.225667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17292159-218b-4eda-96b8-40907c459275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.226334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17292159-218b-4eda-96b8-40907c459275 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.226638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17292159-218b-4eda-96b8-40907c459275 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.226939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17292159-218b-4eda-96b8-40907c459275 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.227621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17292159-218b-4eda-96b8-40907c459275 None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:03.227922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17292159-218b-4eda-96b8-40907c459275 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.229310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17292159-218b-4eda-96b8-40907c459275 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.242367 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 283/578] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:03.243258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17292159-218b-4eda-96b8-40907c459275 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:03.296471 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:03.336756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12da61e-3fb9-4111-addf-3c2aab5c0e8d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:03.349405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:03.356371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17292159-218b-4eda-96b8-40907c459275 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:03.371908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fe26df-a47b-4ce6-9297-3c046bc2e71a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.372542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fe26df-a47b-4ce6-9297-3c046bc2e71a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.372816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fe26df-a47b-4ce6-9297-3c046bc2e71a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.373083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fe26df-a47b-4ce6-9297-3c046bc2e71a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.384680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fe26df-a47b-4ce6-9297-3c046bc2e71a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.386760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fe26df-a47b-4ce6-9297-3c046bc2e71a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.392838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d443a247-869b-43ea-93d7-84502a24977e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.393458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d443a247-869b-43ea-93d7-84502a24977e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.393745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d443a247-869b-43ea-93d7-84502a24977e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.394014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d443a247-869b-43ea-93d7-84502a24977e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-d8fe26df-a47b-4ce6-9297-3c046bc2e71a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:03.403381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:03.404347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:03.405267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:03.405267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:03.405267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:03.406202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 296/579] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:03.410845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fe26df-a47b-4ce6-9297-3c046bc2e71a None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:03.466781 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d443a247-869b-43ea-93d7-84502a24977e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.468432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d443a247-869b-43ea-93d7-84502a24977e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.516764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fe26df-a47b-4ce6-9297-3c046bc2e71a None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:03.533126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:03.555040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ebeb4-3dec-40ad-ae75-b3cedcc415be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.555685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ebeb4-3dec-40ad-ae75-b3cedcc415be None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.555965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ebeb4-3dec-40ad-ae75-b3cedcc415be None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.556236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ebeb4-3dec-40ad-ae75-b3cedcc415be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.563057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:03.568955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241ebeb4-3dec-40ad-ae75-b3cedcc415be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.570474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241ebeb4-3dec-40ad-ae75-b3cedcc415be None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.570937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 284/580] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:17:03.576215 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d443a247-869b-43ea-93d7-84502a24977e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-241ebeb4-3dec-40ad-ae75-b3cedcc415be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:03.585909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:03.587116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:03.587995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:03.587995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:03.587995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:03.587995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 297/581] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:03.587995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ebeb4-3dec-40ad-ae75-b3cedcc415be None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:03.690301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d443a247-869b-43ea-93d7-84502a24977e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:03.700226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ebeb4-3dec-40ad-ae75-b3cedcc415be None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:03.724077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8980f256-7c3e-46a2-a514-022b2ab61a41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.724711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8980f256-7c3e-46a2-a514-022b2ab61a41 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.724988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8980f256-7c3e-46a2-a514-022b2ab61a41 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.725258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8980f256-7c3e-46a2-a514-022b2ab61a41 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.735346 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8980f256-7c3e-46a2-a514-022b2ab61a41 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.736806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8980f256-7c3e-46a2-a514-022b2ab61a41 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.740790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb77c54-302a-4405-bd65-d948a8f1c310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.741646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb77c54-302a-4405-bd65-d948a8f1c310 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.742136 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb77c54-302a-4405-bd65-d948a8f1c310 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.742574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb77c54-302a-4405-bd65-d948a8f1c310 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.743383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb77c54-302a-4405-bd65-d948a8f1c310 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:03.743855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb77c54-302a-4405-bd65-d948a8f1c310 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.746348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb77c54-302a-4405-bd65-d948a8f1c310 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-8980f256-7c3e-46a2-a514-022b2ab61a41 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:03.752282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:03.753207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:03.754088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:03.754088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:03.754088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:03.754088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 285/582] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:03.754354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8980f256-7c3e-46a2-a514-022b2ab61a41 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:03.765831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 298/583] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:03.767215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb77c54-302a-4405-bd65-d948a8f1c310 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:03.798460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:03.813601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:03.874236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb77c54-302a-4405-bd65-d948a8f1c310 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:03.877402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8980f256-7c3e-46a2-a514-022b2ab61a41 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:03.910968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f20375-0545-40e7-84f1-efd46d2b4ac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.911592 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f20375-0545-40e7-84f1-efd46d2b4ac8 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.911864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f20375-0545-40e7-84f1-efd46d2b4ac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.912132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f20375-0545-40e7-84f1-efd46d2b4ac8 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.912770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f20375-0545-40e7-84f1-efd46d2b4ac8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:03.912975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1979cd69-695f-4f83-b94f-2a22c4be9b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:03.913095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f20375-0545-40e7-84f1-efd46d2b4ac8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.913612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1979cd69-695f-4f83-b94f-2a22c4be9b32 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:03.913891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1979cd69-695f-4f83-b94f-2a22c4be9b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:03.914194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1979cd69-695f-4f83-b94f-2a22c4be9b32 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:03.914450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f20375-0545-40e7-84f1-efd46d2b4ac8 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.924984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1979cd69-695f-4f83-b94f-2a22c4be9b32 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:03.926468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1979cd69-695f-4f83-b94f-2a22c4be9b32 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:03.927350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 299/584] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:03.932544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f20375-0545-40e7-84f1-efd46d2b4ac8 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-1979cd69-695f-4f83-b94f-2a22c4be9b32 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:03.941250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:03.942287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:03.943188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:03.943188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:03.943188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:03.943188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 286/585] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:03.943188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1979cd69-695f-4f83-b94f-2a22c4be9b32 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.031553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:04.038930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:04.041816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1979cd69-695f-4f83-b94f-2a22c4be9b32 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.047673 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f20375-0545-40e7-84f1-efd46d2b4ac8 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.075488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707dc177-7c02-4dfc-98a1-8cd1a9c9272c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.076431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707dc177-7c02-4dfc-98a1-8cd1a9c9272c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.077087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707dc177-7c02-4dfc-98a1-8cd1a9c9272c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.077365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707dc177-7c02-4dfc-98a1-8cd1a9c9272c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.083137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d92588-fa7f-48fc-b0bd-0c436ead812a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.083744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d92588-fa7f-48fc-b0bd-0c436ead812a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.084021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d92588-fa7f-48fc-b0bd-0c436ead812a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.084289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d92588-fa7f-48fc-b0bd-0c436ead812a None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.084925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d92588-fa7f-48fc-b0bd-0c436ead812a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:04.085135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707dc177-7c02-4dfc-98a1-8cd1a9c9272c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.085233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d92588-fa7f-48fc-b0bd-0c436ead812a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.086609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d92588-fa7f-48fc-b0bd-0c436ead812a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.086735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707dc177-7c02-4dfc-98a1-8cd1a9c9272c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-707dc177-7c02-4dfc-98a1-8cd1a9c9272c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:04.097817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:04.098805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:04.107485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 287/586] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/users/neutron => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:04.108257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 300/587] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:04.108342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707dc177-7c02-4dfc-98a1-8cd1a9c9272c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.108432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d92588-fa7f-48fc-b0bd-0c436ead812a None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.206226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707dc177-7c02-4dfc-98a1-8cd1a9c9272c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.216625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d92588-fa7f-48fc-b0bd-0c436ead812a None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.242023 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca35254-c500-49c5-8f85-e7417d54c63f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.242699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca35254-c500-49c5-8f85-e7417d54c63f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.243295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca35254-c500-49c5-8f85-e7417d54c63f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.243574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca35254-c500-49c5-8f85-e7417d54c63f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.250565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca35254-c500-49c5-8f85-e7417d54c63f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.252058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca35254-c500-49c5-8f85-e7417d54c63f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.254776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7f4ed2-fde4-4641-878e-728fc64886e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.255548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7f4ed2-fde4-4641-878e-728fc64886e5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.255966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7f4ed2-fde4-4641-878e-728fc64886e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.256376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7f4ed2-fde4-4641-878e-728fc64886e5 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.257147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7f4ed2-fde4-4641-878e-728fc64886e5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:04.257608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7f4ed2-fde4-4641-878e-728fc64886e5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.259147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7f4ed2-fde4-4641-878e-728fc64886e5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-7ca35254-c500-49c5-8f85-e7417d54c63f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:04.264274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:04.265163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:04.271721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:04.271721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:04.271721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:04.271721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:04.271721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:04.271721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:04.271721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:04.271721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:04.271721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 288/588] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:04.271721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca35254-c500-49c5-8f85-e7417d54c63f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.281244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 301/589] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:04.281244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7f4ed2-fde4-4641-878e-728fc64886e5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.295674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:04.297539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:04.378758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca35254-c500-49c5-8f85-e7417d54c63f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.394006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7f4ed2-fde4-4641-878e-728fc64886e5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.415541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7373b49f-0181-4720-96ec-2b3c2c7d3cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.416132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7373b49f-0181-4720-96ec-2b3c2c7d3cf7 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.416830 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7373b49f-0181-4720-96ec-2b3c2c7d3cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.417098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7373b49f-0181-4720-96ec-2b3c2c7d3cf7 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.417874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7373b49f-0181-4720-96ec-2b3c2c7d3cf7 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:04.418208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7373b49f-0181-4720-96ec-2b3c2c7d3cf7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.419647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7373b49f-0181-4720-96ec-2b3c2c7d3cf7 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.430299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be811b-650c-4647-99a9-84b90d828488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.431048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be811b-650c-4647-99a9-84b90d828488 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.431420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be811b-650c-4647-99a9-84b90d828488 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.431800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be811b-650c-4647-99a9-84b90d828488 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.442232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 289/590] 10.222.0.16 () {56 vars in 1139 bytes} [Tue Sep 13 15:17:04 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:04.442232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7373b49f-0181-4720-96ec-2b3c2c7d3cf7 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.444388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26be811b-650c-4647-99a9-84b90d828488 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.445385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26be811b-650c-4647-99a9-84b90d828488 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-26be811b-650c-4647-99a9-84b90d828488 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:04.457202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:04.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 302/591] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:04 2022] GET /identity/v3/users/cinder => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:04.464233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be811b-650c-4647-99a9-84b90d828488 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.532017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:04.538539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:04.555078 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7373b49f-0181-4720-96ec-2b3c2c7d3cf7 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.574489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be811b-650c-4647-99a9-84b90d828488 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.591193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ed9524-bf8c-4770-86ff-6c7449671d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.592143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ed9524-bf8c-4770-86ff-6c7449671d20 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.592143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ed9524-bf8c-4770-86ff-6c7449671d20 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.592679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ed9524-bf8c-4770-86ff-6c7449671d20 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.593360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ed9524-bf8c-4770-86ff-6c7449671d20 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:04.593664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ed9524-bf8c-4770-86ff-6c7449671d20 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.595546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ed9524-bf8c-4770-86ff-6c7449671d20 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.607473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 290/592] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:04.608674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ed9524-bf8c-4770-86ff-6c7449671d20 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.615040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ea9854-123b-4eef-9e69-cb2747d446db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.616790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ea9854-123b-4eef-9e69-cb2747d446db None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.617234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ea9854-123b-4eef-9e69-cb2747d446db None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.617510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ea9854-123b-4eef-9e69-cb2747d446db None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.626360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ea9854-123b-4eef-9e69-cb2747d446db None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.627834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ea9854-123b-4eef-9e69-cb2747d446db None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-f1ea9854-123b-4eef-9e69-cb2747d446db None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:04.640690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:04.641624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:04.642568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:04.642568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:04.642568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:04.642568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:04.642568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:04.642568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:04.642568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:04.642568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:04.646660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 303/593] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:04.647536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ea9854-123b-4eef-9e69-cb2747d446db None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.723506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ed9524-bf8c-4770-86ff-6c7449671d20 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.753866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ea9854-123b-4eef-9e69-cb2747d446db None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.766502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5777cbe7-91d7-4c3d-82dd-d7b5a2a42d3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.767240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5777cbe7-91d7-4c3d-82dd-d7b5a2a42d3a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.767659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5777cbe7-91d7-4c3d-82dd-d7b5a2a42d3a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.767925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5777cbe7-91d7-4c3d-82dd-d7b5a2a42d3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.777621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:04.782261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:04.782736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5777cbe7-91d7-4c3d-82dd-d7b5a2a42d3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.784329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5777cbe7-91d7-4c3d-82dd-d7b5a2a42d3a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.790600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce6ac20-5014-467b-bb87-3d4fcc13d454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.791591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce6ac20-5014-467b-bb87-3d4fcc13d454 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.791950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce6ac20-5014-467b-bb87-3d4fcc13d454 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.792250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce6ac20-5014-467b-bb87-3d4fcc13d454 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.793028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce6ac20-5014-467b-bb87-3d4fcc13d454 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:04.793344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce6ac20-5014-467b-bb87-3d4fcc13d454 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.794838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce6ac20-5014-467b-bb87-3d4fcc13d454 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-5777cbe7-91d7-4c3d-82dd-d7b5a2a42d3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:04.800273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:04.801210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:04.802088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:04.802088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:04.802088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:04.802088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 291/594] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:04.802854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5777cbe7-91d7-4c3d-82dd-d7b5a2a42d3a None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.816157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 304/595] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Sep 13 15:17:04 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:04.817312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce6ac20-5014-467b-bb87-3d4fcc13d454 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:04.916126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5777cbe7-91d7-4c3d-82dd-d7b5a2a42d3a None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.930115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce6ac20-5014-467b-bb87-3d4fcc13d454 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:04.954222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3fff40-0bcb-4827-9d59-c86c5c637f69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.954993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3fff40-0bcb-4827-9d59-c86c5c637f69 None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.955276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3fff40-0bcb-4827-9d59-c86c5c637f69 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.955554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3fff40-0bcb-4827-9d59-c86c5c637f69 None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/2ce646dcb8124685ac7a1c1eca664d04/roles/da0ea75123ba496fb410e87b8df992ba` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.964860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf09cbc-215c-4747-bbbf-049fd009c803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:04.965483 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf09cbc-215c-4747-bbbf-049fd009c803 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:04.965759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf09cbc-215c-4747-bbbf-049fd009c803 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:04.966055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf09cbc-215c-4747-bbbf-049fd009c803 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:04.966777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf09cbc-215c-4747-bbbf-049fd009c803 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:04.967112 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf09cbc-215c-4747-bbbf-049fd009c803 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.968544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf09cbc-215c-4747-bbbf-049fd009c803 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.972501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3fff40-0bcb-4827-9d59-c86c5c637f69 None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=2ce646dcb8124685ac7a1c1eca664d04, role_id=da0ea75123ba496fb410e87b8df992ba)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:04.974202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3fff40-0bcb-4827-9d59-c86c5c637f69 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:04.980228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 305/596] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:04.981059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf09cbc-215c-4747-bbbf-049fd009c803 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:05.006201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 292/597] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:17:04 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/2ce646dcb8124685ac7a1c1eca664d04/roles/da0ea75123ba496fb410e87b8df992ba => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:17:05.007734 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3fff40-0bcb-4827-9d59-c86c5c637f69 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:05.046522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:05.059268 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:05.113167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf09cbc-215c-4747-bbbf-049fd009c803 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:05.149132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3fff40-0bcb-4827-9d59-c86c5c637f69 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:05.160793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1e5531-6f59-4970-ac96-48a9a05bb961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:05.161538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e5531-6f59-4970-ac96-48a9a05bb961 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:05.161888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e5531-6f59-4970-ac96-48a9a05bb961 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:05.162278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e5531-6f59-4970-ac96-48a9a05bb961 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:05.163010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1e5531-6f59-4970-ac96-48a9a05bb961 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:05.163383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1e5531-6f59-4970-ac96-48a9a05bb961 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:05.165025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1e5531-6f59-4970-ac96-48a9a05bb961 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:05.184510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 306/598] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:05.185990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1e5531-6f59-4970-ac96-48a9a05bb961 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:05.201982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff0c4c-00c7-4dbd-9498-4a8f460e0215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:05.203357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff0c4c-00c7-4dbd-9498-4a8f460e0215 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:05.203895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff0c4c-00c7-4dbd-9498-4a8f460e0215 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:05.204364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff0c4c-00c7-4dbd-9498-4a8f460e0215 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:05.220165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ff0c4c-00c7-4dbd-9498-4a8f460e0215 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:05.222419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ff0c4c-00c7-4dbd-9498-4a8f460e0215 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-29ff0c4c-00c7-4dbd-9498-4a8f460e0215 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:05.240262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:05.241390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:05.242558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:05.242558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:05.242558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:05.242558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 293/599] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:05.242558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff0c4c-00c7-4dbd-9498-4a8f460e0215 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:05.302353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1e5531-6f59-4970-ac96-48a9a05bb961 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:05.311219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:05.340332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76018cb8-870d-4998-97eb-f3d6186c9d47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:05.340938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:05.341074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76018cb8-870d-4998-97eb-f3d6186c9d47 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:05.341263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76018cb8-870d-4998-97eb-f3d6186c9d47 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:05.342156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76018cb8-870d-4998-97eb-f3d6186c9d47 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:05.342756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76018cb8-870d-4998-97eb-f3d6186c9d47 None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=35ece665f492449282ab03ad91f0b254) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:05.343110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76018cb8-870d-4998-97eb-f3d6186c9d47 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:05.344657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76018cb8-870d-4998-97eb-f3d6186c9d47 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:05.350208 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 307/600] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:05 2022] GET /identity/v3/role_assignments?user.id=35ece665f492449282ab03ad91f0b254&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:05.351005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76018cb8-870d-4998-97eb-f3d6186c9d47 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:05.352108 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff0c4c-00c7-4dbd-9498-4a8f460e0215 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:05.404490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7ba00b-d6ec-4fb8-9a20-2bc867515180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:05.405274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ba00b-d6ec-4fb8-9a20-2bc867515180 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:05.406445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ba00b-d6ec-4fb8-9a20-2bc867515180 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:05.406749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7ba00b-d6ec-4fb8-9a20-2bc867515180 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:05.416394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7ba00b-d6ec-4fb8-9a20-2bc867515180 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:05.417913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7ba00b-d6ec-4fb8-9a20-2bc867515180 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-5e7ba00b-d6ec-4fb8-9a20-2bc867515180 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:05.430942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:05.431947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:05.432837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:05.432837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:05.432837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:05.432837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:05.432837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:05.432837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:05.432837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:05.432837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:05.437254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 294/601] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:05.438955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d40c59-f12b-414b-8ed3-5a6f65694725 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:05.439246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d40c59-f12b-414b-8ed3-5a6f65694725 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:05.439525 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d40c59-f12b-414b-8ed3-5a6f65694725 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:05.441183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 295/602] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:05.441955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d40c59-f12b-414b-8ed3-5a6f65694725 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:05.482790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76018cb8-870d-4998-97eb-f3d6186c9d47 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:05.522132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c2113d-966c-45e3-90ee-5726e4bee473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:05.522695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c2113d-966c-45e3-90ee-5726e4bee473 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:05.522977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c2113d-966c-45e3-90ee-5726e4bee473 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:05.523254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c2113d-966c-45e3-90ee-5726e4bee473 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:05.523916 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c2113d-966c-45e3-90ee-5726e4bee473 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:05.524223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c2113d-966c-45e3-90ee-5726e4bee473 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:05.525649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c2113d-966c-45e3-90ee-5726e4bee473 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:05.539210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 308/603] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:05.540594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7201eec3-7d1f-4899-aa75-54e7e66be2b1 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:05.540874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7201eec3-7d1f-4899-aa75-54e7e66be2b1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:05.541154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7201eec3-7d1f-4899-aa75-54e7e66be2b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:05.560469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7201eec3-7d1f-4899-aa75-54e7e66be2b1 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:05.561736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7201eec3-7d1f-4899-aa75-54e7e66be2b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:05.571281 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d40c59-f12b-414b-8ed3-5a6f65694725 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:05.589643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:05.617961 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1b1104-e15e-416f-b135-480ca5963ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:05.619065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1b1104-e15e-416f-b135-480ca5963ba8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:05.619406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1b1104-e15e-416f-b135-480ca5963ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:05.619979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1b1104-e15e-416f-b135-480ca5963ba8 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:05.620786 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1b1104-e15e-416f-b135-480ca5963ba8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:05.621389 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1b1104-e15e-416f-b135-480ca5963ba8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:05.623735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1b1104-e15e-416f-b135-480ca5963ba8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:05.638680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 296/604] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:05.640237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1b1104-e15e-416f-b135-480ca5963ba8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:05.657983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:05.660626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 309/605] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:05.661816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7201eec3-7d1f-4899-aa75-54e7e66be2b1 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:05.822060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7201eec3-7d1f-4899-aa75-54e7e66be2b1 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:05.828480 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1b1104-e15e-416f-b135-480ca5963ba8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:05.876640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f777e66-d3fa-4c82-9cba-55e789426afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:05.877618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f777e66-d3fa-4c82-9cba-55e789426afe None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:05.878902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f777e66-d3fa-4c82-9cba-55e789426afe None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:05.879995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f777e66-d3fa-4c82-9cba-55e789426afe None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:05.892618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0fe508-44a0-45ef-a5d9-2e22e501f9cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:05.893595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f777e66-d3fa-4c82-9cba-55e789426afe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:05.894476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0fe508-44a0-45ef-a5d9-2e22e501f9cc None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:05.896364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0fe508-44a0-45ef-a5d9-2e22e501f9cc None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:05.896821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0fe508-44a0-45ef-a5d9-2e22e501f9cc None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/171e916244804add8f18b44b77b03d72/roles/da0ea75123ba496fb410e87b8df992ba` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:05.897643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f777e66-d3fa-4c82-9cba-55e789426afe None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-0f777e66-d3fa-4c82-9cba-55e789426afe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:05.923196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:05.924163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:05.925046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:05.925046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:05.925046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:05.925046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:05.925046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:05.925046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:05.925046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:05.925046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:05.927025 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 297/606] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:05.928967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0fe508-44a0-45ef-a5d9-2e22e501f9cc None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=171e916244804add8f18b44b77b03d72, role_id=da0ea75123ba496fb410e87b8df992ba)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:05.930749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0fe508-44a0-45ef-a5d9-2e22e501f9cc None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:05.933680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f777e66-d3fa-4c82-9cba-55e789426afe None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:05.960300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 310/607] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:17:05 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/171e916244804add8f18b44b77b03d72/roles/da0ea75123ba496fb410e87b8df992ba => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:17:06.035124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:06.042682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0fe508-44a0-45ef-a5d9-2e22e501f9cc None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:06.058564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:06.111143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f777e66-d3fa-4c82-9cba-55e789426afe None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:06.165632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd7d77c-69ed-46be-b1b0-b155b5b8e426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:06.166567 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd7d77c-69ed-46be-b1b0-b155b5b8e426 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:06.166986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd7d77c-69ed-46be-b1b0-b155b5b8e426 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:06.167396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd7d77c-69ed-46be-b1b0-b155b5b8e426 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:06.168228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd7d77c-69ed-46be-b1b0-b155b5b8e426 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:06.168665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd7d77c-69ed-46be-b1b0-b155b5b8e426 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:06.170341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd7d77c-69ed-46be-b1b0-b155b5b8e426 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:06.184236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 298/608] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:06.195111 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd7d77c-69ed-46be-b1b0-b155b5b8e426 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:06.204658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0fe508-44a0-45ef-a5d9-2e22e501f9cc None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:06.249682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4899ee98-b960-44a4-91ac-e26932864b5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:06.250360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4899ee98-b960-44a4-91ac-e26932864b5b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:06.250636 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4899ee98-b960-44a4-91ac-e26932864b5b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:06.250906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4899ee98-b960-44a4-91ac-e26932864b5b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:06.271902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4899ee98-b960-44a4-91ac-e26932864b5b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-4899ee98-b960-44a4-91ac-e26932864b5b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:06.283007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:06.284215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:06.285789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 311/609] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:06.291008 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4899ee98-b960-44a4-91ac-e26932864b5b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:06.362659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd7d77c-69ed-46be-b1b0-b155b5b8e426 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:06.384479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:06.424928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:06.437384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e45984-a7b8-48af-a7f1-8c0acf482035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:06.438304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e45984-a7b8-48af-a7f1-8c0acf482035 None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:06.438585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e45984-a7b8-48af-a7f1-8c0acf482035 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:06.438906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e45984-a7b8-48af-a7f1-8c0acf482035 None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/39c54901c5b5400f96e084fd4adf54e1/roles/da0ea75123ba496fb410e87b8df992ba` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:06.452380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4899ee98-b960-44a4-91ac-e26932864b5b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:06.457655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e45984-a7b8-48af-a7f1-8c0acf482035 None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=39c54901c5b5400f96e084fd4adf54e1, role_id=da0ea75123ba496fb410e87b8df992ba)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:06.459174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e45984-a7b8-48af-a7f1-8c0acf482035 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:06.481225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 299/610] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:17:06 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/39c54901c5b5400f96e084fd4adf54e1/roles/da0ea75123ba496fb410e87b8df992ba => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:17:06.505894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b698494-4cac-4e1a-997c-37669a43c829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:06.506563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b698494-4cac-4e1a-997c-37669a43c829 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:06.506836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b698494-4cac-4e1a-997c-37669a43c829 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:06.507102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b698494-4cac-4e1a-997c-37669a43c829 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:06.507781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b698494-4cac-4e1a-997c-37669a43c829 None admin] RBAC: Adding query filter params (name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:06.508084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b698494-4cac-4e1a-997c-37669a43c829 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:06.509561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b698494-4cac-4e1a-997c-37669a43c829 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:06.520580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 312/611] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:06.526652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e45984-a7b8-48af-a7f1-8c0acf482035 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:06.662304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e45984-a7b8-48af-a7f1-8c0acf482035 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:06.700311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3f7f20-6721-45f2-a114-76f163e5af40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:06.701026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3f7f20-6721-45f2-a114-76f163e5af40 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:06.701384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3f7f20-6721-45f2-a114-76f163e5af40 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:06.701713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3f7f20-6721-45f2-a114-76f163e5af40 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:06.714015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3f7f20-6721-45f2-a114-76f163e5af40 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:06.715658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3f7f20-6721-45f2-a114-76f163e5af40 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-4f3f7f20-6721-45f2-a114-76f163e5af40 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:06.732044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:06.733080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:06.733979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:06.733979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:06.733979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:06.733979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 300/612] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:06.740799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b698494-4cac-4e1a-997c-37669a43c829 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:06.856785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b698494-4cac-4e1a-997c-37669a43c829 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:06.888107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:06.895157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2f3a17-f944-4eaa-ad2a-2bb71b84ab43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:06.895782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f3a17-f944-4eaa-ad2a-2bb71b84ab43 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:06.896063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f3a17-f944-4eaa-ad2a-2bb71b84ab43 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:06.896355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f3a17-f944-4eaa-ad2a-2bb71b84ab43 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:06.896981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2f3a17-f944-4eaa-ad2a-2bb71b84ab43 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:06.897279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2f3a17-f944-4eaa-ad2a-2bb71b84ab43 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:06.898686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2f3a17-f944-4eaa-ad2a-2bb71b84ab43 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:06.911970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 313/613] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:06.917427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3f7f20-6721-45f2-a114-76f163e5af40 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:06.974328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:07.072877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3f7f20-6721-45f2-a114-76f163e5af40 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:07.107413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b018fabf-93b0-4430-ac32-85836610e7c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:07.108087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b018fabf-93b0-4430-ac32-85836610e7c1 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:07.108419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b018fabf-93b0-4430-ac32-85836610e7c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:07.108687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b018fabf-93b0-4430-ac32-85836610e7c1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:07.116231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b018fabf-93b0-4430-ac32-85836610e7c1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:07.117696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b018fabf-93b0-4430-ac32-85836610e7c1 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-b018fabf-93b0-4430-ac32-85836610e7c1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:07.135999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:07.137323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:07.138574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 301/614] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Sep 13 15:17:06 2022] GET /identity/v3/users/nova => generated 82 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:07.145225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2f3a17-f944-4eaa-ad2a-2bb71b84ab43 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:07.287913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2f3a17-f944-4eaa-ad2a-2bb71b84ab43 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:07.333856 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbab0777-5787-450c-8579-9c788c4d913c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:07.334509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbab0777-5787-450c-8579-9c788c4d913c None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:07.334785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbab0777-5787-450c-8579-9c788c4d913c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:07.335062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbab0777-5787-450c-8579-9c788c4d913c None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:07.335798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbab0777-5787-450c-8579-9c788c4d913c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:07.336102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbab0777-5787-450c-8579-9c788c4d913c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:07.337538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbab0777-5787-450c-8579-9c788c4d913c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:07.361675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 314/615] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Sep 13 15:17:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:07.367637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b018fabf-93b0-4430-ac32-85836610e7c1 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:07.507301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:07.514860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b018fabf-93b0-4430-ac32-85836610e7c1 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:07.551659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd8ceee-04de-4b5d-9ede-a8765be4d1a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:07.552289 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd8ceee-04de-4b5d-9ede-a8765be4d1a2 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:07.552563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd8ceee-04de-4b5d-9ede-a8765be4d1a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:07.552830 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd8ceee-04de-4b5d-9ede-a8765be4d1a2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:07.630423 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd8ceee-04de-4b5d-9ede-a8765be4d1a2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:07.631981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd8ceee-04de-4b5d-9ede-a8765be4d1a2 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:07.720581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7ba33c-c6dc-4444-b645-7cafe2a9c76f None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:07.720938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7ba33c-c6dc-4444-b645-7cafe2a9c76f None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:07.721134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7ba33c-c6dc-4444-b645-7cafe2a9c76f None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:07.722780 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 315/616] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:07.726713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:07.727896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcc79f2-f12f-4792-a244-64d6605d4717 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:07.728178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcc79f2-f12f-4792-a244-64d6605d4717 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:07.728461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcc79f2-f12f-4792-a244-64d6605d4717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:07.735128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 302/617] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:17:07.741726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd8ceee-04de-4b5d-9ede-a8765be4d1a2 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:07.769558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-3dcc79f2-f12f-4792-a244-64d6605d4717 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:07.771995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3dcc79f2-f12f-4792-a244-64d6605d4717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:07.858793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:07.868434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 316/618] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:17:07.917817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd8ceee-04de-4b5d-9ede-a8765be4d1a2 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:07.958824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793cf268-1b4f-4342-9260-f42d5313f0bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:07.959485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793cf268-1b4f-4342-9260-f42d5313f0bf None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:07.959828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793cf268-1b4f-4342-9260-f42d5313f0bf None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:07.960104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793cf268-1b4f-4342-9260-f42d5313f0bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:07.972406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793cf268-1b4f-4342-9260-f42d5313f0bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:07.974077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793cf268-1b4f-4342-9260-f42d5313f0bf None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-793cf268-1b4f-4342-9260-f42d5313f0bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:07.991137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:07.992175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:07.993803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:07.993803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:07.993803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:07.993803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 303/619] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:07.998501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcc79f2-f12f-4792-a244-64d6605d4717 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:08.091244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6fa7a0-0f1f-4b5e-b1e6-95c43ccb0d28 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.091991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6fa7a0-0f1f-4b5e-b1e6-95c43ccb0d28 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.092912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6fa7a0-0f1f-4b5e-b1e6-95c43ccb0d28 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.095989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 304/620] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:08.099526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcc79f2-f12f-4792-a244-64d6605d4717 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:08.100815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb846f-de36-4538-9d0d-65284d0eabef None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.101345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb846f-de36-4538-9d0d-65284d0eabef None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.101828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbb846f-de36-4538-9d0d-65284d0eabef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.131173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-ecbb846f-de36-4538-9d0d-65284d0eabef None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:08.132999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ecbb846f-de36-4538-9d0d-65284d0eabef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:08.141884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3633607-05b1-49db-a657-e5fa5166fac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:08.146894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3633607-05b1-49db-a657-e5fa5166fac8 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.147176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3633607-05b1-49db-a657-e5fa5166fac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.147449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3633607-05b1-49db-a657-e5fa5166fac8 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.148156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3633607-05b1-49db-a657-e5fa5166fac8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:08.148465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3633607-05b1-49db-a657-e5fa5166fac8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:08.149869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3633607-05b1-49db-a657-e5fa5166fac8 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:08.176103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 317/621] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:08.182380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3633607-05b1-49db-a657-e5fa5166fac8 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:08.214004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:08.247737 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:08.250750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 305/622] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:17:08.251901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbb846f-de36-4538-9d0d-65284d0eabef None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:08.372356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbb846f-de36-4538-9d0d-65284d0eabef None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:08.388111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3633607-05b1-49db-a657-e5fa5166fac8 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:08.442927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335348b2-2fbc-469b-8412-496ffeb1b3d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:08.445018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335348b2-2fbc-469b-8412-496ffeb1b3d7 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.446617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335348b2-2fbc-469b-8412-496ffeb1b3d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.447041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335348b2-2fbc-469b-8412-496ffeb1b3d7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.459453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed0a951-ea01-49e0-b691-9e3174ee8b10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:08.461278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed0a951-ea01-49e0-b691-9e3174ee8b10 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.466352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335348b2-2fbc-469b-8412-496ffeb1b3d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:08.466494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed0a951-ea01-49e0-b691-9e3174ee8b10 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.466793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed0a951-ea01-49e0-b691-9e3174ee8b10 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-335348b2-2fbc-469b-8412-496ffeb1b3d7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:08.472103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:08.473267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:08.474354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:08.477072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 306/623] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:08.483516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335348b2-2fbc-469b-8412-496ffeb1b3d7 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:08.484187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed0a951-ea01-49e0-b691-9e3174ee8b10 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:08.485668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed0a951-ea01-49e0-b691-9e3174ee8b10 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-aed0a951-ea01-49e0-b691-9e3174ee8b10 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:08.506842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:08.507883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:08.508773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:08.508773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:08.508773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:08.508773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:08.508773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:08.508773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:08.508773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:08.508773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:08.508773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 318/624] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:08.519229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed0a951-ea01-49e0-b691-9e3174ee8b10 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:08.550472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:08.616662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335348b2-2fbc-469b-8412-496ffeb1b3d7 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:08.667190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ce0623-7c0f-4026-a5ce-7ebaeeacd4d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:08.668173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ce0623-7c0f-4026-a5ce-7ebaeeacd4d6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.668464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ce0623-7c0f-4026-a5ce-7ebaeeacd4d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.668995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ce0623-7c0f-4026-a5ce-7ebaeeacd4d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.669671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ce0623-7c0f-4026-a5ce-7ebaeeacd4d6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:08.670097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ce0623-7c0f-4026-a5ce-7ebaeeacd4d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:08.672016 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ce0623-7c0f-4026-a5ce-7ebaeeacd4d6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:08.682621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 307/625] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:08.683584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ce0623-7c0f-4026-a5ce-7ebaeeacd4d6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:08.687885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:08.693078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed0a951-ea01-49e0-b691-9e3174ee8b10 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:08.729611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bde6c4-59c6-40fe-89d2-4c04dadd5f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:08.730256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bde6c4-59c6-40fe-89d2-4c04dadd5f46 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.730534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bde6c4-59c6-40fe-89d2-4c04dadd5f46 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.730800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bde6c4-59c6-40fe-89d2-4c04dadd5f46 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.731487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bde6c4-59c6-40fe-89d2-4c04dadd5f46 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:08.731794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bde6c4-59c6-40fe-89d2-4c04dadd5f46 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:08.733273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bde6c4-59c6-40fe-89d2-4c04dadd5f46 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:08.746381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 319/626] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:08.752300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fffe5d-1156-4793-a1ca-cabc14c97de7 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.752583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fffe5d-1156-4793-a1ca-cabc14c97de7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.752874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fffe5d-1156-4793-a1ca-cabc14c97de7 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.754285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 320/627] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:08 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:08.754840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fffe5d-1156-4793-a1ca-cabc14c97de7 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:08.783491 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:08.787838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ce0623-7c0f-4026-a5ce-7ebaeeacd4d6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:08.823889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bed33e-e62d-4d65-8895-554cd7240b4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:08.824534 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bed33e-e62d-4d65-8895-554cd7240b4d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.824813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bed33e-e62d-4d65-8895-554cd7240b4d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.825085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bed33e-e62d-4d65-8895-554cd7240b4d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.835436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bed33e-e62d-4d65-8895-554cd7240b4d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-91bed33e-e62d-4d65-8895-554cd7240b4d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:08.840587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:08.841499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:08.842466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 308/628] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:08.843178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bed33e-e62d-4d65-8895-554cd7240b4d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:08.867478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fffe5d-1156-4793-a1ca-cabc14c97de7 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:08.902235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7776c45d-a51e-4ed7-a4b5-eeacb5e72fbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:08.902793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776c45d-a51e-4ed7-a4b5-eeacb5e72fbe None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.903067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776c45d-a51e-4ed7-a4b5-eeacb5e72fbe None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.903334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776c45d-a51e-4ed7-a4b5-eeacb5e72fbe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.915062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7776c45d-a51e-4ed7-a4b5-eeacb5e72fbe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:08.916587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7776c45d-a51e-4ed7-a4b5-eeacb5e72fbe None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-7776c45d-a51e-4ed7-a4b5-eeacb5e72fbe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:08.933855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:08.935007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:08.936016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:08.936016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:08.936016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:08.936016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 321/629] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:08.936417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6514f1c9-856d-4d6c-92d2-3c10e924ab2a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.936495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6514f1c9-856d-4d6c-92d2-3c10e924ab2a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.937038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6514f1c9-856d-4d6c-92d2-3c10e924ab2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.952677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bed33e-e62d-4d65-8895-554cd7240b4d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:08.955388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-6514f1c9-856d-4d6c-92d2-3c10e924ab2a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:08.956587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6514f1c9-856d-4d6c-92d2-3c10e924ab2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:08.964264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:08.990396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23c5e5c-84af-4e71-9109-01241ecb048e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:08.991014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23c5e5c-84af-4e71-9109-01241ecb048e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:08.991403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23c5e5c-84af-4e71-9109-01241ecb048e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:08.991680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23c5e5c-84af-4e71-9109-01241ecb048e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:08.992405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23c5e5c-84af-4e71-9109-01241ecb048e None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=f027e913f53447a988f74ae4dff84b2c) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:08.992712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23c5e5c-84af-4e71-9109-01241ecb048e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:08.994189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23c5e5c-84af-4e71-9109-01241ecb048e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:09.000412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 309/630] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:08 2022] GET /identity/v3/role_assignments?user.id=f027e913f53447a988f74ae4dff84b2c&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:09.000972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23c5e5c-84af-4e71-9109-01241ecb048e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:09.036095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 322/631] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:09.037427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6514f1c9-856d-4d6c-92d2-3c10e924ab2a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:09.064555 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:09.197400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23c5e5c-84af-4e71-9109-01241ecb048e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:09.215499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6514f1c9-856d-4d6c-92d2-3c10e924ab2a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:09.239355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54855f4-08ea-46d4-a2d1-d7882b89a60e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:09.239976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54855f4-08ea-46d4-a2d1-d7882b89a60e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:09.240260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54855f4-08ea-46d4-a2d1-d7882b89a60e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:09.240532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54855f4-08ea-46d4-a2d1-d7882b89a60e None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:09.241197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54855f4-08ea-46d4-a2d1-d7882b89a60e None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:09.241499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54855f4-08ea-46d4-a2d1-d7882b89a60e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:09.242918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54855f4-08ea-46d4-a2d1-d7882b89a60e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:09.256224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 310/632] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:09.256224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7a4a35-86d4-4e6d-bb68-beb330375e95 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:09.256224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7a4a35-86d4-4e6d-bb68-beb330375e95 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:09.256224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7a4a35-86d4-4e6d-bb68-beb330375e95 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:09.256224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 311/633] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:09 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:09.261674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119afbb5-dd49-4ffe-ba99-4c200d6eaee8 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:09.261956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119afbb5-dd49-4ffe-ba99-4c200d6eaee8 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:09.262269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119afbb5-dd49-4ffe-ba99-4c200d6eaee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:09.266719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9261f6b1-ad93-4161-90a9-de2d680fa1b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:09.267878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261f6b1-ad93-4161-90a9-de2d680fa1b6 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:09.268314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261f6b1-ad93-4161-90a9-de2d680fa1b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:09.268587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9261f6b1-ad93-4161-90a9-de2d680fa1b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:09.269411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9261f6b1-ad93-4161-90a9-de2d680fa1b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:09.270449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9261f6b1-ad93-4161-90a9-de2d680fa1b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:09.271844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9261f6b1-ad93-4161-90a9-de2d680fa1b6 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:09.283866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-119afbb5-dd49-4ffe-ba99-4c200d6eaee8 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:09.285071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-119afbb5-dd49-4ffe-ba99-4c200d6eaee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:09.286615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 323/634] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:09.289379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9261f6b1-ad93-4161-90a9-de2d680fa1b6 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:09.292649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:09.374987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 312/635] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:09.375930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119afbb5-dd49-4ffe-ba99-4c200d6eaee8 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:09.391609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:09.412640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9261f6b1-ad93-4161-90a9-de2d680fa1b6 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:09.455572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea7c95d-d205-4112-a351-03bd6e9c733b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:09.456794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea7c95d-d205-4112-a351-03bd6e9c733b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:09.457387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea7c95d-d205-4112-a351-03bd6e9c733b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:09.457869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea7c95d-d205-4112-a351-03bd6e9c733b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:09.472253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea7c95d-d205-4112-a351-03bd6e9c733b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:09.474281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea7c95d-d205-4112-a351-03bd6e9c733b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:09.476482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119afbb5-dd49-4ffe-ba99-4c200d6eaee8 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-2ea7c95d-d205-4112-a351-03bd6e9c733b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:09.493190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:09.494228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:09.495159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:09.495159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:09.495159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:09.498182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 324/636] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:09.499212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea7c95d-d205-4112-a351-03bd6e9c733b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:09.518651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0663521-25db-488e-85b3-00375a677f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:09.519326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0663521-25db-488e-85b3-00375a677f4a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:09.519594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0663521-25db-488e-85b3-00375a677f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:09.519856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0663521-25db-488e-85b3-00375a677f4a None admin] PATH_INFO: `/v3/users/glance` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:09.527576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0663521-25db-488e-85b3-00375a677f4a None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:09.529077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0663521-25db-488e-85b3-00375a677f4a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-c0663521-25db-488e-85b3-00375a677f4a None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:09.544128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:09.545219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:09.546227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 313/637] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:09 2022] GET /identity/v3/users/glance => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:09.550907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0663521-25db-488e-85b3-00375a677f4a None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:09.571285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:09.628437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea7c95d-d205-4112-a351-03bd6e9c733b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:09.629801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:09.657076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0663521-25db-488e-85b3-00375a677f4a None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:09.668939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be9fc0d-2b45-4758-b33f-149af5daf8ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:09.669692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be9fc0d-2b45-4758-b33f-149af5daf8ac None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:09.670083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be9fc0d-2b45-4758-b33f-149af5daf8ac None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:09.670389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be9fc0d-2b45-4758-b33f-149af5daf8ac None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:09.681102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be9fc0d-2b45-4758-b33f-149af5daf8ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-1be9fc0d-2b45-4758-b33f-149af5daf8ac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:09.686447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:09.687386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:09.688229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 325/638] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:09.688928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be9fc0d-2b45-4758-b33f-149af5daf8ac None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:09.698815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc66d747-8110-4c3e-bf63-fa52922c9114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:09.699515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc66d747-8110-4c3e-bf63-fa52922c9114 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:09.699824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc66d747-8110-4c3e-bf63-fa52922c9114 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:09.700062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc66d747-8110-4c3e-bf63-fa52922c9114 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:09.700802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc66d747-8110-4c3e-bf63-fa52922c9114 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:09.701204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc66d747-8110-4c3e-bf63-fa52922c9114 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:09.702619 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc66d747-8110-4c3e-bf63-fa52922c9114 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:09.718900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 314/639] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:09.720106 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc66d747-8110-4c3e-bf63-fa52922c9114 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:09.800978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be9fc0d-2b45-4758-b33f-149af5daf8ac None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:09.816044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:09.827596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc66d747-8110-4c3e-bf63-fa52922c9114 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:09.837325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4350b040-f168-487e-9cb0-ccd205b91961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:09.837941 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4350b040-f168-487e-9cb0-ccd205b91961 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:09.838247 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4350b040-f168-487e-9cb0-ccd205b91961 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:09.838499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4350b040-f168-487e-9cb0-ccd205b91961 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:09.839224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4350b040-f168-487e-9cb0-ccd205b91961 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:09.839526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4350b040-f168-487e-9cb0-ccd205b91961 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:09.840933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4350b040-f168-487e-9cb0-ccd205b91961 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:09.863963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 326/640] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Sep 13 15:17:09 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:09.864174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971f5c29-afff-460d-8fbd-bfc943ab7569 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:09.864662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4350b040-f168-487e-9cb0-ccd205b91961 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:09.864778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971f5c29-afff-460d-8fbd-bfc943ab7569 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:09.864986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971f5c29-afff-460d-8fbd-bfc943ab7569 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:09.865261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971f5c29-afff-460d-8fbd-bfc943ab7569 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:09.867414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:09.879351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971f5c29-afff-460d-8fbd-bfc943ab7569 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-971f5c29-afff-460d-8fbd-bfc943ab7569 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:09.884110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:09.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 315/641] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:09.886081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971f5c29-afff-460d-8fbd-bfc943ab7569 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:09.975982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4350b040-f168-487e-9cb0-ccd205b91961 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:09.989868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971f5c29-afff-460d-8fbd-bfc943ab7569 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:10.018559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8978c593-3596-44e9-b72a-2aec247d64af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.019302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8978c593-3596-44e9-b72a-2aec247d64af None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.019583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8978c593-3596-44e9-b72a-2aec247d64af None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.019963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8978c593-3596-44e9-b72a-2aec247d64af None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.020730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8978c593-3596-44e9-b72a-2aec247d64af None admin] RBAC: Adding query filter params (name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:10.021043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8978c593-3596-44e9-b72a-2aec247d64af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.022491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8978c593-3596-44e9-b72a-2aec247d64af None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.025285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45ed936-69be-4563-9c85-e286638db563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.025925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45ed936-69be-4563-9c85-e286638db563 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.026239 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45ed936-69be-4563-9c85-e286638db563 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.026512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45ed936-69be-4563-9c85-e286638db563 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.036902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 327/642] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:10.036902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8978c593-3596-44e9-b72a-2aec247d64af None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:10.037227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45ed936-69be-4563-9c85-e286638db563 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.037227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45ed936-69be-4563-9c85-e286638db563 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-d45ed936-69be-4563-9c85-e286638db563 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:10.051211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:10.052318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 316/643] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:09 2022] GET /identity/v3/users/swift => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:10.053244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45ed936-69be-4563-9c85-e286638db563 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:10.083216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:10.112580 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:10.154522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8978c593-3596-44e9-b72a-2aec247d64af None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:10.160413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45ed936-69be-4563-9c85-e286638db563 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:10.188245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6f13ec-6e4c-40b8-99ab-b5529127658c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.188855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6f13ec-6e4c-40b8-99ab-b5529127658c None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.189267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6f13ec-6e4c-40b8-99ab-b5529127658c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.189540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6f13ec-6e4c-40b8-99ab-b5529127658c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.194380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0125e3-9721-4eb3-a4a9-7f4b3dd95d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.195015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0125e3-9721-4eb3-a4a9-7f4b3dd95d25 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.195254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0125e3-9721-4eb3-a4a9-7f4b3dd95d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.195519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0125e3-9721-4eb3-a4a9-7f4b3dd95d25 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.196176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0125e3-9721-4eb3-a4a9-7f4b3dd95d25 None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:10.196483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0125e3-9721-4eb3-a4a9-7f4b3dd95d25 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.197882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0125e3-9721-4eb3-a4a9-7f4b3dd95d25 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.206678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 317/644] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:10.207492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0125e3-9721-4eb3-a4a9-7f4b3dd95d25 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:10.270022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6f13ec-6e4c-40b8-99ab-b5529127658c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.271434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6f13ec-6e4c-40b8-99ab-b5529127658c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.307810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:10.311631 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0125e3-9721-4eb3-a4a9-7f4b3dd95d25 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:10.328850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:10.347963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065ae909-a124-4e7b-b940-1ba40f57c5ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.348695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065ae909-a124-4e7b-b940-1ba40f57c5ee None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.348971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065ae909-a124-4e7b-b940-1ba40f57c5ee None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.349260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065ae909-a124-4e7b-b940-1ba40f57c5ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.360419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065ae909-a124-4e7b-b940-1ba40f57c5ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.361875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065ae909-a124-4e7b-b940-1ba40f57c5ee None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-065ae909-a124-4e7b-b940-1ba40f57c5ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:10.376560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:10.377523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:10.378614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:10.378614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:10.378614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:10.378614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 318/645] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:10.378614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065ae909-a124-4e7b-b940-1ba40f57c5ee None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:10.385955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 328/646] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:17:10.391258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6f13ec-6e4c-40b8-99ab-b5529127658c None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:10.484566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065ae909-a124-4e7b-b940-1ba40f57c5ee None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:10.497535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6f13ec-6e4c-40b8-99ab-b5529127658c None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:10.520369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58583a2b-01c3-4fee-b5ed-85b5b306dce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.520919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58583a2b-01c3-4fee-b5ed-85b5b306dce4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.521199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58583a2b-01c3-4fee-b5ed-85b5b306dce4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.521470 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58583a2b-01c3-4fee-b5ed-85b5b306dce4 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.522224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58583a2b-01c3-4fee-b5ed-85b5b306dce4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:10.522532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58583a2b-01c3-4fee-b5ed-85b5b306dce4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.523976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58583a2b-01c3-4fee-b5ed-85b5b306dce4 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.533912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38411ef2-78ff-4e67-bfbd-2f60444a8dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.534794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38411ef2-78ff-4e67-bfbd-2f60444a8dd2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.535259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38411ef2-78ff-4e67-bfbd-2f60444a8dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.535710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38411ef2-78ff-4e67-bfbd-2f60444a8dd2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.538331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:10.546125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 319/647] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:10.546125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58583a2b-01c3-4fee-b5ed-85b5b306dce4 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:10.549733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38411ef2-78ff-4e67-bfbd-2f60444a8dd2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.552357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38411ef2-78ff-4e67-bfbd-2f60444a8dd2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-38411ef2-78ff-4e67-bfbd-2f60444a8dd2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:10.568384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:10.569321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:10.571176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:10.571176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:10.571176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:10.571176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 329/648] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:10.571176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38411ef2-78ff-4e67-bfbd-2f60444a8dd2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:10.580482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:10.666239 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58583a2b-01c3-4fee-b5ed-85b5b306dce4 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:10.682599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38411ef2-78ff-4e67-bfbd-2f60444a8dd2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:10.701514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4920552a-d571-49ef-890b-441fcefeb4d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.702285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4920552a-d571-49ef-890b-441fcefeb4d4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.702652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4920552a-d571-49ef-890b-441fcefeb4d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.703008 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4920552a-d571-49ef-890b-441fcefeb4d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.703736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4920552a-d571-49ef-890b-441fcefeb4d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:10.704240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4920552a-d571-49ef-890b-441fcefeb4d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.705699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4920552a-d571-49ef-890b-441fcefeb4d4 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.721578 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 320/649] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:10.721578 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4920552a-d571-49ef-890b-441fcefeb4d4 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:10.724411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a92c72-d8f9-463e-9d9f-37308eaf9683 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.725140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a92c72-d8f9-463e-9d9f-37308eaf9683 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.725490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a92c72-d8f9-463e-9d9f-37308eaf9683 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.725831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a92c72-d8f9-463e-9d9f-37308eaf9683 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.738042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a92c72-d8f9-463e-9d9f-37308eaf9683 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.739868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a92c72-d8f9-463e-9d9f-37308eaf9683 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-b1a92c72-d8f9-463e-9d9f-37308eaf9683 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:10.754750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:10.755698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:10.756574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:10.756574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:10.756574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:10.756574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 330/650] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:10.756794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a92c72-d8f9-463e-9d9f-37308eaf9683 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:10.804913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:10.818755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:10.833983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4920552a-d571-49ef-890b-441fcefeb4d4 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:10.866518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d76a5d-137c-45df-a624-81fc4c80a7b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.867245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d76a5d-137c-45df-a624-81fc4c80a7b2 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.867489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d76a5d-137c-45df-a624-81fc4c80a7b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.867776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d76a5d-137c-45df-a624-81fc4c80a7b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.873392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a92c72-d8f9-463e-9d9f-37308eaf9683 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:10.878938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d76a5d-137c-45df-a624-81fc4c80a7b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.880396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d76a5d-137c-45df-a624-81fc4c80a7b2 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-f2d76a5d-137c-45df-a624-81fc4c80a7b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:10.894899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:10.895877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:10.896758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:10.896758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:10.896758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:10.896758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 321/651] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:10.897506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d76a5d-137c-45df-a624-81fc4c80a7b2 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:10.915256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d021b5-aec8-43c1-bf5f-57285b568292 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:10.915879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d021b5-aec8-43c1-bf5f-57285b568292 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:10.916155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d021b5-aec8-43c1-bf5f-57285b568292 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:10.916430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d021b5-aec8-43c1-bf5f-57285b568292 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:10.917076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d021b5-aec8-43c1-bf5f-57285b568292 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:10.917380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d021b5-aec8-43c1-bf5f-57285b568292 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:10.918876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d021b5-aec8-43c1-bf5f-57285b568292 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:10.931523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 331/652] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:10.932280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d021b5-aec8-43c1-bf5f-57285b568292 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:11.022353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d76a5d-137c-45df-a624-81fc4c80a7b2 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.038433 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:11.043261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d021b5-aec8-43c1-bf5f-57285b568292 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.061228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4f25f4-4fbf-4292-bb8d-e1de16111ba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.061677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4f25f4-4fbf-4292-bb8d-e1de16111ba3 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.061949 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4f25f4-4fbf-4292-bb8d-e1de16111ba3 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.062240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4f25f4-4fbf-4292-bb8d-e1de16111ba3 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.069676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4f25f4-4fbf-4292-bb8d-e1de16111ba3 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.071140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4f25f4-4fbf-4292-bb8d-e1de16111ba3 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.072871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:11.082926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecbea60-9a81-49c8-9fe1-72c3af74482e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.083736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecbea60-9a81-49c8-9fe1-72c3af74482e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-5a4f25f4-4fbf-4292-bb8d-e1de16111ba3 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:11.083871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:11.084931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 322/653] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/users/neutron => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:11.085906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4f25f4-4fbf-4292-bb8d-e1de16111ba3 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:11.086950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecbea60-9a81-49c8-9fe1-72c3af74482e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.087228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecbea60-9a81-49c8-9fe1-72c3af74482e None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.087888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecbea60-9a81-49c8-9fe1-72c3af74482e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:11.088198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecbea60-9a81-49c8-9fe1-72c3af74482e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.089730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecbea60-9a81-49c8-9fe1-72c3af74482e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.102863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 332/654] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:11.103623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecbea60-9a81-49c8-9fe1-72c3af74482e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:11.198280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4f25f4-4fbf-4292-bb8d-e1de16111ba3 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.221549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecbea60-9a81-49c8-9fe1-72c3af74482e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.242521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12885154-0130-433a-adb7-7937c069e037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.243300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12885154-0130-433a-adb7-7937c069e037 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.243586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12885154-0130-433a-adb7-7937c069e037 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.243858 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12885154-0130-433a-adb7-7937c069e037 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.244531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12885154-0130-433a-adb7-7937c069e037 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:11.244833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12885154-0130-433a-adb7-7937c069e037 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.246237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12885154-0130-433a-adb7-7937c069e037 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.263184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 323/655] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:11.263184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12885154-0130-433a-adb7-7937c069e037 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:11.274024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28964d1-dbec-49fc-8803-3f79c36a7940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.274872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28964d1-dbec-49fc-8803-3f79c36a7940 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.275154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28964d1-dbec-49fc-8803-3f79c36a7940 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.275772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28964d1-dbec-49fc-8803-3f79c36a7940 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.285738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28964d1-dbec-49fc-8803-3f79c36a7940 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.287368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28964d1-dbec-49fc-8803-3f79c36a7940 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-b28964d1-dbec-49fc-8803-3f79c36a7940 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:11.306188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:11.307231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 333/656] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:11 2022] GET /identity/v3/users/cinder => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:11.308664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28964d1-dbec-49fc-8803-3f79c36a7940 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:11.335364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:11.337437 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:11.386772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12885154-0130-433a-adb7-7937c069e037 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.417904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28964d1-dbec-49fc-8803-3f79c36a7940 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.421633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0576691d-8aaf-4b9a-a848-d13885cfdfcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.422285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0576691d-8aaf-4b9a-a848-d13885cfdfcd None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.422563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0576691d-8aaf-4b9a-a848-d13885cfdfcd None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.422838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0576691d-8aaf-4b9a-a848-d13885cfdfcd None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.423595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0576691d-8aaf-4b9a-a848-d13885cfdfcd None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:11.423900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0576691d-8aaf-4b9a-a848-d13885cfdfcd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.425351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0576691d-8aaf-4b9a-a848-d13885cfdfcd None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.442452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 324/657] 10.222.0.16 () {56 vars in 1139 bytes} [Tue Sep 13 15:17:11 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:11.443460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0576691d-8aaf-4b9a-a848-d13885cfdfcd None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:11.450853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80564f40-7f25-40f8-8517-ff4376d12552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.451553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80564f40-7f25-40f8-8517-ff4376d12552 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.451825 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80564f40-7f25-40f8-8517-ff4376d12552 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.452096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80564f40-7f25-40f8-8517-ff4376d12552 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.458807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80564f40-7f25-40f8-8517-ff4376d12552 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.460254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80564f40-7f25-40f8-8517-ff4376d12552 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-80564f40-7f25-40f8-8517-ff4376d12552 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:11.471288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:11.472350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:11.473418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:11.473418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:11.473418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:11.473418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:11.473418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:11.473418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:11.473418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:11.473418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:11.473418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 334/658] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:11.473418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80564f40-7f25-40f8-8517-ff4376d12552 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:11.544747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0576691d-8aaf-4b9a-a848-d13885cfdfcd None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.553912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:11.561722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:11.576342 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80564f40-7f25-40f8-8517-ff4376d12552 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.579153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00a893d-a826-43a9-a7e8-9522211da66a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.579797 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a893d-a826-43a9-a7e8-9522211da66a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.580074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a893d-a826-43a9-a7e8-9522211da66a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.580347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a893d-a826-43a9-a7e8-9522211da66a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.587144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00a893d-a826-43a9-a7e8-9522211da66a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.588558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00a893d-a826-43a9-a7e8-9522211da66a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-d00a893d-a826-43a9-a7e8-9522211da66a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:11.599671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:11.600680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 325/659] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e922890-ca1d-4f07-bea7-ffc81f6dc123 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.606695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e922890-ca1d-4f07-bea7-ffc81f6dc123 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.607579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e922890-ca1d-4f07-bea7-ffc81f6dc123 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.607579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 326/660] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:11 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:11.607579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e922890-ca1d-4f07-bea7-ffc81f6dc123 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:11.611920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3d95c6-b75c-4a72-9427-28e325df1445 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.612534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3d95c6-b75c-4a72-9427-28e325df1445 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.612816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3d95c6-b75c-4a72-9427-28e325df1445 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.613074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3d95c6-b75c-4a72-9427-28e325df1445 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.614176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3d95c6-b75c-4a72-9427-28e325df1445 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:11.614480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3d95c6-b75c-4a72-9427-28e325df1445 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.615949 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3d95c6-b75c-4a72-9427-28e325df1445 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.633473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 335/661] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Sep 13 15:17:11 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:11.634330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3d95c6-b75c-4a72-9427-28e325df1445 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:11.714798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e922890-ca1d-4f07-bea7-ffc81f6dc123 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.749558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3d95c6-b75c-4a72-9427-28e325df1445 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.750764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e62461c-f032-4ad5-ac1c-552a47962081 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.751384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e62461c-f032-4ad5-ac1c-552a47962081 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.751665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e62461c-f032-4ad5-ac1c-552a47962081 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.751934 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e62461c-f032-4ad5-ac1c-552a47962081 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.791842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5254126-1eba-4016-8636-ba1d07f9c046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.792460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5254126-1eba-4016-8636-ba1d07f9c046 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.792739 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5254126-1eba-4016-8636-ba1d07f9c046 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.793602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5254126-1eba-4016-8636-ba1d07f9c046 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.794343 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5254126-1eba-4016-8636-ba1d07f9c046 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:11.794651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5254126-1eba-4016-8636-ba1d07f9c046 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.796969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5254126-1eba-4016-8636-ba1d07f9c046 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.799084 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:11.799757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:11.809244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 336/662] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:11.810127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5254126-1eba-4016-8636-ba1d07f9c046 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:11.828883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e62461c-f032-4ad5-ac1c-552a47962081 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.830496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e62461c-f032-4ad5-ac1c-552a47962081 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.910167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5254126-1eba-4016-8636-ba1d07f9c046 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:11.928307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 327/663] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:17:11.929331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec91aaeb-8ca4-49e2-bfd5-fb5d20eb7b38 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.929610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec91aaeb-8ca4-49e2-bfd5-fb5d20eb7b38 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.929881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec91aaeb-8ca4-49e2-bfd5-fb5d20eb7b38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.946951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9797fa-d24e-4110-9aaf-a5a73e6c95e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:11.947393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-ec91aaeb-8ca4-49e2-bfd5-fb5d20eb7b38 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:11.947578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9797fa-d24e-4110-9aaf-a5a73e6c95e8 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:11.947850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9797fa-d24e-4110-9aaf-a5a73e6c95e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:11.948109 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9797fa-d24e-4110-9aaf-a5a73e6c95e8 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:11.948621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ec91aaeb-8ca4-49e2-bfd5-fb5d20eb7b38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:11.948790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9797fa-d24e-4110-9aaf-a5a73e6c95e8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:11.949089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9797fa-d24e-4110-9aaf-a5a73e6c95e8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:11.950506 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9797fa-d24e-4110-9aaf-a5a73e6c95e8 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:11.968797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 337/664] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:11.969624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9797fa-d24e-4110-9aaf-a5a73e6c95e8 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:12.017301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 328/665] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:12.018202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec91aaeb-8ca4-49e2-bfd5-fb5d20eb7b38 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:12.026781 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:12.039191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:12.089419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9797fa-d24e-4110-9aaf-a5a73e6c95e8 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:12.125581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973bdf4f-279b-4a66-920d-7db9548dfba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:12.126369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973bdf4f-279b-4a66-920d-7db9548dfba2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:12.126937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973bdf4f-279b-4a66-920d-7db9548dfba2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:12.127222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973bdf4f-279b-4a66-920d-7db9548dfba2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:12.132174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec91aaeb-8ca4-49e2-bfd5-fb5d20eb7b38 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:12.165320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070413ed-e7a1-4d34-ae78-160a3a726d67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:12.165851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070413ed-e7a1-4d34-ae78-160a3a726d67 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:12.166160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070413ed-e7a1-4d34-ae78-160a3a726d67 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:12.166433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070413ed-e7a1-4d34-ae78-160a3a726d67 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:12.167156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070413ed-e7a1-4d34-ae78-160a3a726d67 None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=2ce646dcb8124685ac7a1c1eca664d04) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:12.167463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070413ed-e7a1-4d34-ae78-160a3a726d67 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:12.168905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070413ed-e7a1-4d34-ae78-160a3a726d67 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:12.186145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 329/666] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:12 2022] GET /identity/v3/role_assignments?user.id=2ce646dcb8124685ac7a1c1eca664d04&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:12.186145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070413ed-e7a1-4d34-ae78-160a3a726d67 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:12.211117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973bdf4f-279b-4a66-920d-7db9548dfba2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:12.212913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973bdf4f-279b-4a66-920d-7db9548dfba2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:12.271120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:12.279195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:12.302394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070413ed-e7a1-4d34-ae78-160a3a726d67 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:12.327931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 338/667] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:17:12.328867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973bdf4f-279b-4a66-920d-7db9548dfba2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:12.341632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c86a4b-e5dc-4ba6-8d23-709c67596adb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:12.342348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c86a4b-e5dc-4ba6-8d23-709c67596adb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:12.342680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c86a4b-e5dc-4ba6-8d23-709c67596adb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:12.342976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c86a4b-e5dc-4ba6-8d23-709c67596adb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:12.354435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c86a4b-e5dc-4ba6-8d23-709c67596adb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:12.355956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c86a4b-e5dc-4ba6-8d23-709c67596adb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-63c86a4b-e5dc-4ba6-8d23-709c67596adb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:12.376808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:12.377834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:12.378801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:12.378801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:12.378801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:12.378801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 330/668] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:12.378801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c86a4b-e5dc-4ba6-8d23-709c67596adb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:12.461823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973bdf4f-279b-4a66-920d-7db9548dfba2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:12.517156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765c6de1-f9a4-4c10-8726-491e1d28be58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:12.517156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765c6de1-f9a4-4c10-8726-491e1d28be58 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:12.517156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765c6de1-f9a4-4c10-8726-491e1d28be58 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:12.517156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765c6de1-f9a4-4c10-8726-491e1d28be58 None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/35ece665f492449282ab03ad91f0b254/roles/da0ea75123ba496fb410e87b8df992ba` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:12.534136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c86a4b-e5dc-4ba6-8d23-709c67596adb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:12.543839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765c6de1-f9a4-4c10-8726-491e1d28be58 None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=35ece665f492449282ab03ad91f0b254, role_id=da0ea75123ba496fb410e87b8df992ba)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:12.545377 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765c6de1-f9a4-4c10-8726-491e1d28be58 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:12.560196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:12.574130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 339/669] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:17:12 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/35ece665f492449282ab03ad91f0b254/roles/da0ea75123ba496fb410e87b8df992ba => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:17:12.574130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765c6de1-f9a4-4c10-8726-491e1d28be58 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:12.592689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:12.597279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e104fdc-99f9-4e03-a5bd-fa4c0f174d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:12.599189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e104fdc-99f9-4e03-a5bd-fa4c0f174d7a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:12.599189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e104fdc-99f9-4e03-a5bd-fa4c0f174d7a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:12.599189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e104fdc-99f9-4e03-a5bd-fa4c0f174d7a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:12.634131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e104fdc-99f9-4e03-a5bd-fa4c0f174d7a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-9e104fdc-99f9-4e03-a5bd-fa4c0f174d7a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:12.637413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:12.638412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:12.639586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 331/670] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:12.640350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e104fdc-99f9-4e03-a5bd-fa4c0f174d7a None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:12.784102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765c6de1-f9a4-4c10-8726-491e1d28be58 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:12.815443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e104fdc-99f9-4e03-a5bd-fa4c0f174d7a None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:12.838654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a031e-7bb0-4ed3-8a8e-aee2be3e42ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:12.839062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a031e-7bb0-4ed3-8a8e-aee2be3e42ce None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:12.839407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a031e-7bb0-4ed3-8a8e-aee2be3e42ce None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:12.839739 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070a031e-7bb0-4ed3-8a8e-aee2be3e42ce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:12.850914 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070a031e-7bb0-4ed3-8a8e-aee2be3e42ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:12.852458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070a031e-7bb0-4ed3-8a8e-aee2be3e42ce None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:12.865481 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d80285-a150-43e3-940b-3692eb55ac43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:12.866409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d80285-a150-43e3-940b-3692eb55ac43 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:12.866891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d80285-a150-43e3-940b-3692eb55ac43 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:12.867339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d80285-a150-43e3-940b-3692eb55ac43 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-070a031e-7bb0-4ed3-8a8e-aee2be3e42ce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:12.867628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:12.868539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:12.869458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:12.869458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:12.869458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:12.869458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 340/671] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:12.869906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a031e-7bb0-4ed3-8a8e-aee2be3e42ce None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:12.870678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d80285-a150-43e3-940b-3692eb55ac43 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:12.871112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d80285-a150-43e3-940b-3692eb55ac43 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:12.872673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d80285-a150-43e3-940b-3692eb55ac43 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:12.886811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 332/672] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:12.890191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d80285-a150-43e3-940b-3692eb55ac43 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:12.933364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:12.939753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:12.975945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070a031e-7bb0-4ed3-8a8e-aee2be3e42ce None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:13.004233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d80285-a150-43e3-940b-3692eb55ac43 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:13.011110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d7aadc-378c-43ed-9745-e3e5055cdf8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:13.011854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d7aadc-378c-43ed-9745-e3e5055cdf8e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:13.012224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d7aadc-378c-43ed-9745-e3e5055cdf8e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:13.012588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d7aadc-378c-43ed-9745-e3e5055cdf8e None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:13.013339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d7aadc-378c-43ed-9745-e3e5055cdf8e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:13.013807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d7aadc-378c-43ed-9745-e3e5055cdf8e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:13.015321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d7aadc-378c-43ed-9745-e3e5055cdf8e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:13.026146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 341/673] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:13.026146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d7aadc-378c-43ed-9745-e3e5055cdf8e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:13.046951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a198596-4a1e-4c7e-ac39-a04e9d3d062d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:13.047963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a198596-4a1e-4c7e-ac39-a04e9d3d062d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:13.048754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a198596-4a1e-4c7e-ac39-a04e9d3d062d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:13.049183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a198596-4a1e-4c7e-ac39-a04e9d3d062d None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:13.050252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a198596-4a1e-4c7e-ac39-a04e9d3d062d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:13.050692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a198596-4a1e-4c7e-ac39-a04e9d3d062d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:13.052677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a198596-4a1e-4c7e-ac39-a04e9d3d062d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:13.071643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 333/674] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:13.071643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a198596-4a1e-4c7e-ac39-a04e9d3d062d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:13.146969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d7aadc-378c-43ed-9745-e3e5055cdf8e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:13.172719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:13.177748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a198596-4a1e-4c7e-ac39-a04e9d3d062d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:13.180176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:13.191196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271df227-1cc6-4563-927f-be2a2745796b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:13.192286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271df227-1cc6-4563-927f-be2a2745796b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:13.192656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271df227-1cc6-4563-927f-be2a2745796b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:13.193294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271df227-1cc6-4563-927f-be2a2745796b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:13.203519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271df227-1cc6-4563-927f-be2a2745796b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:13.206003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271df227-1cc6-4563-927f-be2a2745796b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-271df227-1cc6-4563-927f-be2a2745796b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:13.218683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:13.219772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:13.220718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:13.220718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:13.220718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:13.220718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:13.220718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:13.220718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:13.220718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:13.220718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:13.221139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9785ac49-b90e-4388-9bde-a946fc2dee34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:13.221139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9785ac49-b90e-4388-9bde-a946fc2dee34 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:13.221139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9785ac49-b90e-4388-9bde-a946fc2dee34 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:13.221139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9785ac49-b90e-4388-9bde-a946fc2dee34 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:13.221409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 342/675] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:13.226059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271df227-1cc6-4563-927f-be2a2745796b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:13.238844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9785ac49-b90e-4388-9bde-a946fc2dee34 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:13.240334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9785ac49-b90e-4388-9bde-a946fc2dee34 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-9785ac49-b90e-4388-9bde-a946fc2dee34 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:13.255732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:13.258085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:13.260163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:13.260163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:13.260163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:13.260163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 334/676] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:13.260163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9785ac49-b90e-4388-9bde-a946fc2dee34 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:13.344814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271df227-1cc6-4563-927f-be2a2745796b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:13.364163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9785ac49-b90e-4388-9bde-a946fc2dee34 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:13.382707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d27817b-a181-4204-ad4a-87cecf21bb61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:13.383380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d27817b-a181-4204-ad4a-87cecf21bb61 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:13.383742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d27817b-a181-4204-ad4a-87cecf21bb61 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:13.384231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d27817b-a181-4204-ad4a-87cecf21bb61 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:13.397063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d27817b-a181-4204-ad4a-87cecf21bb61 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:13.398557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d27817b-a181-4204-ad4a-87cecf21bb61 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:13.400357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8f0bb1-4e9c-4284-9c38-dfd67d997d6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:13.400984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8f0bb1-4e9c-4284-9c38-dfd67d997d6b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:13.401265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8f0bb1-4e9c-4284-9c38-dfd67d997d6b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:13.401539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8f0bb1-4e9c-4284-9c38-dfd67d997d6b None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:13.402262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8f0bb1-4e9c-4284-9c38-dfd67d997d6b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:13.402567 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8f0bb1-4e9c-4284-9c38-dfd67d997d6b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:13.404092 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8f0bb1-4e9c-4284-9c38-dfd67d997d6b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-9d27817b-a181-4204-ad4a-87cecf21bb61 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:13.410935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:13.412873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:13.415499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:13.415499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:13.415499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:13.415499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:13.415499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:13.415499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:13.415499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:13.415499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:13.416934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 343/677] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:13.422394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d27817b-a181-4204-ad4a-87cecf21bb61 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:13.422710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 335/678] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:13.426158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8f0bb1-4e9c-4284-9c38-dfd67d997d6b None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:13.439135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:13.468140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:13.536498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8f0bb1-4e9c-4284-9c38-dfd67d997d6b None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:13.552425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d27817b-a181-4204-ad4a-87cecf21bb61 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:13.572026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df49322-f9ab-4c7f-b452-23e8625dfaeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:13.572745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df49322-f9ab-4c7f-b452-23e8625dfaeb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:13.573093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df49322-f9ab-4c7f-b452-23e8625dfaeb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:13.573430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df49322-f9ab-4c7f-b452-23e8625dfaeb None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:13.574218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df49322-f9ab-4c7f-b452-23e8625dfaeb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:13.574589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df49322-f9ab-4c7f-b452-23e8625dfaeb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:13.576161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df49322-f9ab-4c7f-b452-23e8625dfaeb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:13.589712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 336/679] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:13.590869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df49322-f9ab-4c7f-b452-23e8625dfaeb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:13.603138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1a75ee-cd13-444a-90f9-041228d07358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:13.604323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1a75ee-cd13-444a-90f9-041228d07358 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:13.604641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1a75ee-cd13-444a-90f9-041228d07358 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:13.604913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1a75ee-cd13-444a-90f9-041228d07358 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:13.605968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1a75ee-cd13-444a-90f9-041228d07358 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:13.606573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1a75ee-cd13-444a-90f9-041228d07358 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:13.608589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1a75ee-cd13-444a-90f9-041228d07358 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:13.625591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 344/680] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:13.626820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1a75ee-cd13-444a-90f9-041228d07358 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:13.693897 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:13.719682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df49322-f9ab-4c7f-b452-23e8625dfaeb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:13.756579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb60502-4c56-4ba7-8e05-9e7903a9e681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:13.757190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb60502-4c56-4ba7-8e05-9e7903a9e681 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:13.757468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb60502-4c56-4ba7-8e05-9e7903a9e681 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:13.757737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb60502-4c56-4ba7-8e05-9e7903a9e681 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:13.758491 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb60502-4c56-4ba7-8e05-9e7903a9e681 None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=171e916244804add8f18b44b77b03d72) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:13.758796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb60502-4c56-4ba7-8e05-9e7903a9e681 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:13.760295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb60502-4c56-4ba7-8e05-9e7903a9e681 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:13.761434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:13.766055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1a75ee-cd13-444a-90f9-041228d07358 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:13.774699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 337/681] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:13 2022] GET /identity/v3/role_assignments?user.id=171e916244804add8f18b44b77b03d72&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:13.775707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb60502-4c56-4ba7-8e05-9e7903a9e681 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:13.811908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb5b539-ae19-4141-9634-1d9f4057042f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:13.812952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb5b539-ae19-4141-9634-1d9f4057042f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:13.813623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb5b539-ae19-4141-9634-1d9f4057042f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:13.814137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb5b539-ae19-4141-9634-1d9f4057042f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:13.817493 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb5b539-ae19-4141-9634-1d9f4057042f None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=39c54901c5b5400f96e084fd4adf54e1) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:13.819057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb5b539-ae19-4141-9634-1d9f4057042f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:13.823846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb5b539-ae19-4141-9634-1d9f4057042f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:13.836287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 345/682] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:13 2022] GET /identity/v3/role_assignments?user.id=39c54901c5b5400f96e084fd4adf54e1&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 608 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:13.960531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb60502-4c56-4ba7-8e05-9e7903a9e681 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:14.012398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfff1be-1fe9-4142-8bc7-5d657f626d01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:14.013073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfff1be-1fe9-4142-8bc7-5d657f626d01 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:14.013348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfff1be-1fe9-4142-8bc7-5d657f626d01 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:14.013711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfff1be-1fe9-4142-8bc7-5d657f626d01 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:14.023536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfff1be-1fe9-4142-8bc7-5d657f626d01 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:14.025214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfff1be-1fe9-4142-8bc7-5d657f626d01 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:14.027265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-dcfff1be-1fe9-4142-8bc7-5d657f626d01 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:14.044524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:14.045645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:14.046597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 338/683] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Sep 13 15:17:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:14.053608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb5b539-ae19-4141-9634-1d9f4057042f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:14.177401 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb5b539-ae19-4141-9634-1d9f4057042f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:14.214607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a502b7d0-37ec-4003-b416-e4d6d47fa4f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:14.215584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a502b7d0-37ec-4003-b416-e4d6d47fa4f8 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:14.216100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a502b7d0-37ec-4003-b416-e4d6d47fa4f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:14.216619 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a502b7d0-37ec-4003-b416-e4d6d47fa4f8 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:14.218023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a502b7d0-37ec-4003-b416-e4d6d47fa4f8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:14.218559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a502b7d0-37ec-4003-b416-e4d6d47fa4f8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:14.220176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a502b7d0-37ec-4003-b416-e4d6d47fa4f8 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:14.241082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 346/684] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Sep 13 15:17:14 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:14.247017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfff1be-1fe9-4142-8bc7-5d657f626d01 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:14.399570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfff1be-1fe9-4142-8bc7-5d657f626d01 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:14.454869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c45619-a677-42ae-b377-b7cc9abbbea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:14.455504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c45619-a677-42ae-b377-b7cc9abbbea6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:14.455785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c45619-a677-42ae-b377-b7cc9abbbea6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:14.456060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c45619-a677-42ae-b377-b7cc9abbbea6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:14.483689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41d00a2-a8a7-42e1-b60c-529b24824dec None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:14.483984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41d00a2-a8a7-42e1-b60c-529b24824dec None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:14.484170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41d00a2-a8a7-42e1-b60c-529b24824dec None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:14.487049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 347/685] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:14.491768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf4b948-3285-4f1e-9402-23f47da93cdf None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:14.492053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf4b948-3285-4f1e-9402-23f47da93cdf None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:14.492328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf4b948-3285-4f1e-9402-23f47da93cdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:14.512791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7bf4b948-3285-4f1e-9402-23f47da93cdf None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:14.514054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7bf4b948-3285-4f1e-9402-23f47da93cdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:14.549473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:14.568099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:14.574621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c45619-a677-42ae-b377-b7cc9abbbea6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:14.576188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c45619-a677-42ae-b377-b7cc9abbbea6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:14.637883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 348/686] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:14.728417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 339/687] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:17:14.733902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf4b948-3285-4f1e-9402-23f47da93cdf None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:14.869151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf4b948-3285-4f1e-9402-23f47da93cdf None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:14.912260 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7ed9e7-05f8-495c-9674-e73a275e6f7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:14.912911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7ed9e7-05f8-495c-9674-e73a275e6f7a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:14.913189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7ed9e7-05f8-495c-9674-e73a275e6f7a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:14.913513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7ed9e7-05f8-495c-9674-e73a275e6f7a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:14.924055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7ed9e7-05f8-495c-9674-e73a275e6f7a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:14.925519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7ed9e7-05f8-495c-9674-e73a275e6f7a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:14.937089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c45619-a677-42ae-b377-b7cc9abbbea6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-aa7ed9e7-05f8-495c-9674-e73a275e6f7a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:14.953212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:14.954241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:14.955120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:14.955120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:14.955120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:14.956903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 349/688] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:14.962685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7ed9e7-05f8-495c-9674-e73a275e6f7a None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:14.970453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:15.117452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:15.124100 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c45619-a677-42ae-b377-b7cc9abbbea6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:15.142375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7ed9e7-05f8-495c-9674-e73a275e6f7a None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:15.174779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107d4808-2945-4db5-9bd8-26ae642e32e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:15.175420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107d4808-2945-4db5-9bd8-26ae642e32e5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:15.175700 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107d4808-2945-4db5-9bd8-26ae642e32e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:15.175971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107d4808-2945-4db5-9bd8-26ae642e32e5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:15.182538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54b9ec6-5ac9-47a2-bd84-70d50ade174a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:15.184258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54b9ec6-5ac9-47a2-bd84-70d50ade174a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:15.184751 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54b9ec6-5ac9-47a2-bd84-70d50ade174a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:15.185765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54b9ec6-5ac9-47a2-bd84-70d50ade174a None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:15.186921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54b9ec6-5ac9-47a2-bd84-70d50ade174a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:15.187424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54b9ec6-5ac9-47a2-bd84-70d50ade174a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:15.187527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107d4808-2945-4db5-9bd8-26ae642e32e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:15.188926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107d4808-2945-4db5-9bd8-26ae642e32e5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:15.189199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54b9ec6-5ac9-47a2-bd84-70d50ade174a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:15.203050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 350/689] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:15.204817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea637751-7472-4d9e-9bc0-d7978142b349 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-107d4808-2945-4db5-9bd8-26ae642e32e5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:15.205227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:15.206298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:15.207182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:15.207182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:15.207182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:15.207354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea637751-7472-4d9e-9bc0-d7978142b349 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:15.207354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea637751-7472-4d9e-9bc0-d7978142b349 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:15.209190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 340/691] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:15.210231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107d4808-2945-4db5-9bd8-26ae642e32e5 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:15.212552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 351/691] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:15.213754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010fa749-919f-444d-8d1e-7dcc3827adbc None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:15.214122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010fa749-919f-444d-8d1e-7dcc3827adbc None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:15.214823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010fa749-919f-444d-8d1e-7dcc3827adbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:15.240264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-010fa749-919f-444d-8d1e-7dcc3827adbc None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:15.241811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-010fa749-919f-444d-8d1e-7dcc3827adbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:15.315034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:15.326621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 352/692] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:15.327510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010fa749-919f-444d-8d1e-7dcc3827adbc None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:15.358907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107d4808-2945-4db5-9bd8-26ae642e32e5 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:15.424233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3c9e35-0d61-47a9-9782-d650c85fc822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:15.424880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3c9e35-0d61-47a9-9782-d650c85fc822 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:15.425160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3c9e35-0d61-47a9-9782-d650c85fc822 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:15.425434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3c9e35-0d61-47a9-9782-d650c85fc822 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:15.434720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3c9e35-0d61-47a9-9782-d650c85fc822 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:15.436316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3c9e35-0d61-47a9-9782-d650c85fc822 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-9f3c9e35-0d61-47a9-9782-d650c85fc822 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:15.456253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:15.457241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:15.458164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:15.458164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:15.458164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:15.458164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:15.458164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:15.458164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:15.458164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:15.458164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:15.466861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 341/693] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:15.470458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3c9e35-0d61-47a9-9782-d650c85fc822 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:15.493009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:15.508311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010fa749-919f-444d-8d1e-7dcc3827adbc None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:15.548007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dcde22-7112-4e76-8a9c-029ecc7807ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:15.549062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dcde22-7112-4e76-8a9c-029ecc7807ab None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:15.549539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dcde22-7112-4e76-8a9c-029ecc7807ab None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:15.550004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dcde22-7112-4e76-8a9c-029ecc7807ab None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:15.550872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dcde22-7112-4e76-8a9c-029ecc7807ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:15.551371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dcde22-7112-4e76-8a9c-029ecc7807ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:15.553349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dcde22-7112-4e76-8a9c-029ecc7807ab None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:15.567113 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 353/694] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:15.572102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dcde22-7112-4e76-8a9c-029ecc7807ab None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:15.666471 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:15.681838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3c9e35-0d61-47a9-9782-d650c85fc822 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:15.723219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cc8266-0d81-4f66-ad10-a43635a40cb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:15.723813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cc8266-0d81-4f66-ad10-a43635a40cb7 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:15.724098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cc8266-0d81-4f66-ad10-a43635a40cb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:15.724429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cc8266-0d81-4f66-ad10-a43635a40cb7 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:15.725130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cc8266-0d81-4f66-ad10-a43635a40cb7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:15.725436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cc8266-0d81-4f66-ad10-a43635a40cb7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:15.726922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cc8266-0d81-4f66-ad10-a43635a40cb7 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:15.727601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dcde22-7112-4e76-8a9c-029ecc7807ab None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:15.740530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 342/695] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:15.741383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cc8266-0d81-4f66-ad10-a43635a40cb7 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:15.768597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3fa8f2-e1df-4be4-a37b-0d56ca76c270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:15.769622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3fa8f2-e1df-4be4-a37b-0d56ca76c270 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:15.770168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3fa8f2-e1df-4be4-a37b-0d56ca76c270 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:15.770884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3fa8f2-e1df-4be4-a37b-0d56ca76c270 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:15.772147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3fa8f2-e1df-4be4-a37b-0d56ca76c270 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:15.774140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3fa8f2-e1df-4be4-a37b-0d56ca76c270 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:15.802515 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 354/696] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:17:15 2022] POST /identity/v3/users => generated 254 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:17:15.803698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3fa8f2-e1df-4be4-a37b-0d56ca76c270 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:15.823918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:15.859647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cc8266-0d81-4f66-ad10-a43635a40cb7 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:15.897037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d813ec1-c7ac-49b4-ae43-a50586bd171d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:15.897687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d813ec1-c7ac-49b4-ae43-a50586bd171d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:15.897963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d813ec1-c7ac-49b4-ae43-a50586bd171d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:15.898257 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d813ec1-c7ac-49b4-ae43-a50586bd171d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:15.909702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d813ec1-c7ac-49b4-ae43-a50586bd171d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-3d813ec1-c7ac-49b4-ae43-a50586bd171d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:15.914694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:15.915701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:15.917011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 343/697] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:15.922125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d813ec1-c7ac-49b4-ae43-a50586bd171d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:15.922335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3fa8f2-e1df-4be4-a37b-0d56ca76c270 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:15.933895 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:15.962502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc4fbe0-b5e0-458a-adc6-8a02a7197de2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:15.963494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc4fbe0-b5e0-458a-adc6-8a02a7197de2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:15.963773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc4fbe0-b5e0-458a-adc6-8a02a7197de2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:15.964104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc4fbe0-b5e0-458a-adc6-8a02a7197de2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:15.964852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc4fbe0-b5e0-458a-adc6-8a02a7197de2 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=f027e913f53447a988f74ae4dff84b2c) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:15.965160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc4fbe0-b5e0-458a-adc6-8a02a7197de2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:15.966741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc4fbe0-b5e0-458a-adc6-8a02a7197de2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:15.973508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 355/698] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:15 2022] GET /identity/v3/role_assignments?user.id=f027e913f53447a988f74ae4dff84b2c&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=22819414415942d399db534708fe7103 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:16.047753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d813ec1-c7ac-49b4-ae43-a50586bd171d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:16.083166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:16.089681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a23167-a89b-49d1-9974-d54c9cec27ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:16.090328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a23167-a89b-49d1-9974-d54c9cec27ba None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:16.090678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a23167-a89b-49d1-9974-d54c9cec27ba None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:16.090948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a23167-a89b-49d1-9974-d54c9cec27ba None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:16.091818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a23167-a89b-49d1-9974-d54c9cec27ba None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:16.092356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a23167-a89b-49d1-9974-d54c9cec27ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:16.094030 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a23167-a89b-49d1-9974-d54c9cec27ba None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:16.104006 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 344/699] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:16.110221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc4fbe0-b5e0-458a-adc6-8a02a7197de2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:16.225404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc4fbe0-b5e0-458a-adc6-8a02a7197de2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:16.261745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0e1f99-40b1-483a-a0ae-984fb2858dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:16.262461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0e1f99-40b1-483a-a0ae-984fb2858dff None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:16.262734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0e1f99-40b1-483a-a0ae-984fb2858dff None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:16.263000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0e1f99-40b1-483a-a0ae-984fb2858dff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:16.275819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0e1f99-40b1-483a-a0ae-984fb2858dff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:16.277275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0e1f99-40b1-483a-a0ae-984fb2858dff None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-2f0e1f99-40b1-483a-a0ae-984fb2858dff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:16.292546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:16.293634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:16.294742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:16.294742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:16.294742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:16.294742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 356/700] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:16.300562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a23167-a89b-49d1-9974-d54c9cec27ba None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:16.415999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a23167-a89b-49d1-9974-d54c9cec27ba None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:16.454248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a777f18d-c4d3-4d56-882a-f9ccd87c14c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:16.454921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a777f18d-c4d3-4d56-882a-f9ccd87c14c4 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:16.455198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a777f18d-c4d3-4d56-882a-f9ccd87c14c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:16.455468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a777f18d-c4d3-4d56-882a-f9ccd87c14c4 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:16.456162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a777f18d-c4d3-4d56-882a-f9ccd87c14c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:16.456464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a777f18d-c4d3-4d56-882a-f9ccd87c14c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:16.457849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a777f18d-c4d3-4d56-882a-f9ccd87c14c4 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:16.472558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 345/701] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:16.478140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0e1f99-40b1-483a-a0ae-984fb2858dff None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:16.510507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:16.596504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0e1f99-40b1-483a-a0ae-984fb2858dff None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:16.625061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3b514c-36d3-45c9-917b-17df987df447 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:16.625328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3b514c-36d3-45c9-917b-17df987df447 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:16.625803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3b514c-36d3-45c9-917b-17df987df447 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:16.629890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 346/702] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:16.638127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aeae09-cbf9-4758-8cfc-60bf11e2c769 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:16.638127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aeae09-cbf9-4758-8cfc-60bf11e2c769 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:16.638127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2aeae09-cbf9-4758-8cfc-60bf11e2c769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:16.654034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d2aeae09-cbf9-4758-8cfc-60bf11e2c769 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:16.655585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d2aeae09-cbf9-4758-8cfc-60bf11e2c769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:16.672345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1c80ce-1e72-4da8-b822-84d3d72fdb29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:16.673270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1c80ce-1e72-4da8-b822-84d3d72fdb29 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:16.673701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1c80ce-1e72-4da8-b822-84d3d72fdb29 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:16.674130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1c80ce-1e72-4da8-b822-84d3d72fdb29 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:16.687220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1c80ce-1e72-4da8-b822-84d3d72fdb29 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:16.688915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1c80ce-1e72-4da8-b822-84d3d72fdb29 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-9b1c80ce-1e72-4da8-b822-84d3d72fdb29 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:16.703738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:16.704981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:16.705930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:16.705930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:16.705930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:16.705930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:16.705930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:16.705930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:16.705930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:16.705930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:16.705930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:16.705930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:16.706488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 357/703] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:16 2022] GET /identity/v3/users/swift => generated 83 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:16.718924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:16.719100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1c80ce-1e72-4da8-b822-84d3d72fdb29 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:16.746529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 347/704] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:17:16.747842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5784689-0828-449f-bdbf-a4853dce608a None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:16.748119 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5784689-0828-449f-bdbf-a4853dce608a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:16.748453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5784689-0828-449f-bdbf-a4853dce608a None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:16.750206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 348/705] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:16.755256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85d0c8e-d5de-4a7c-a381-f9bfd79aa290 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:16.755780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85d0c8e-d5de-4a7c-a381-f9bfd79aa290 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:16.756058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85d0c8e-d5de-4a7c-a381-f9bfd79aa290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:16.774486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-b85d0c8e-d5de-4a7c-a381-f9bfd79aa290 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:16.775683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b85d0c8e-d5de-4a7c-a381-f9bfd79aa290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:16.868868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 349/706] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:16.879205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:16.902477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1c80ce-1e72-4da8-b822-84d3d72fdb29 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:16.951985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc27f79-6094-44cb-b0f7-f68c908f08ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:16.952667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc27f79-6094-44cb-b0f7-f68c908f08ed None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:16.952944 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc27f79-6094-44cb-b0f7-f68c908f08ed None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:16.953262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc27f79-6094-44cb-b0f7-f68c908f08ed None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:16.954008 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc27f79-6094-44cb-b0f7-f68c908f08ed None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:16.954343 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc27f79-6094-44cb-b0f7-f68c908f08ed None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:16.955848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc27f79-6094-44cb-b0f7-f68c908f08ed None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:16.978534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 358/707] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Sep 13 15:17:16 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:16.984165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85d0c8e-d5de-4a7c-a381-f9bfd79aa290 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:17.127031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85d0c8e-d5de-4a7c-a381-f9bfd79aa290 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:17.140706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:17.175855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9accf4-8f3e-4a2a-b111-a6c9aa735241 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:17.176486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9accf4-8f3e-4a2a-b111-a6c9aa735241 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:17.176765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9accf4-8f3e-4a2a-b111-a6c9aa735241 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:17.177039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9accf4-8f3e-4a2a-b111-a6c9aa735241 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:17.177601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc27f79-6094-44cb-b0f7-f68c908f08ed None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:17.284681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9accf4-8f3e-4a2a-b111-a6c9aa735241 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:17.286193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9accf4-8f3e-4a2a-b111-a6c9aa735241 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:17.287621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc27f79-6094-44cb-b0f7-f68c908f08ed None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:17.325506 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be898a8-3465-49d6-ada6-fa5f234b0bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:17.326699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be898a8-3465-49d6-ada6-fa5f234b0bce None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:17.327173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be898a8-3465-49d6-ada6-fa5f234b0bce None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:17.327646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be898a8-3465-49d6-ada6-fa5f234b0bce None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:17.328426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be898a8-3465-49d6-ada6-fa5f234b0bce None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:17.330032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be898a8-3465-49d6-ada6-fa5f234b0bce None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-3be898a8-3465-49d6-ada6-fa5f234b0bce None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:17.341078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:17.342621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:17:17.343470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:17.350304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 359/708] 10.222.0.16 () {56 vars in 1129 bytes} [Tue Sep 13 15:17:17 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:17.360162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be898a8-3465-49d6-ada6-fa5f234b0bce None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:17.373029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:17.449046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 350/709] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:17:17.450379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9accf4-8f3e-4a2a-b111-a6c9aa735241 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:17.465495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:17.474943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be898a8-3465-49d6-ada6-fa5f234b0bce None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:17.511777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed1803d-c478-48e0-b00b-68b7181d65ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:17.512489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed1803d-c478-48e0-b00b-68b7181d65ee None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:17.512826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed1803d-c478-48e0-b00b-68b7181d65ee None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:17.513161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed1803d-c478-48e0-b00b-68b7181d65ee None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:17.514557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed1803d-c478-48e0-b00b-68b7181d65ee None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:17.514671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed1803d-c478-48e0-b00b-68b7181d65ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:17.516137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed1803d-c478-48e0-b00b-68b7181d65ee None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:17.525731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 360/710] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Sep 13 15:17:17 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:17.525731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed1803d-c478-48e0-b00b-68b7181d65ee None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:17.570720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9accf4-8f3e-4a2a-b111-a6c9aa735241 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:17.593618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:17.609826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449fc77e-eb4b-47db-a0ac-df9b0df5c163 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:17.610773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449fc77e-eb4b-47db-a0ac-df9b0df5c163 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:17.611222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449fc77e-eb4b-47db-a0ac-df9b0df5c163 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:17.611636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449fc77e-eb4b-47db-a0ac-df9b0df5c163 None admin] PATH_INFO: `/v3/services/network` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:17.612365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449fc77e-eb4b-47db-a0ac-df9b0df5c163 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:17.614355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449fc77e-eb4b-47db-a0ac-df9b0df5c163 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:17.625449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed1803d-c478-48e0-b00b-68b7181d65ee None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-449fc77e-eb4b-47db-a0ac-df9b0df5c163 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:17.626451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:17.627630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:17:17.628622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:17.635568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 351/711] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:17 2022] GET /identity/v3/services/network => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:17.636655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449fc77e-eb4b-47db-a0ac-df9b0df5c163 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:17.661548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0072cc-b1b9-4d2f-8800-4f93d3edac7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:17.662532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0072cc-b1b9-4d2f-8800-4f93d3edac7c None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:17.662748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0072cc-b1b9-4d2f-8800-4f93d3edac7c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:17.663089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0072cc-b1b9-4d2f-8800-4f93d3edac7c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:17.674452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0072cc-b1b9-4d2f-8800-4f93d3edac7c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:17.676081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0072cc-b1b9-4d2f-8800-4f93d3edac7c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-2e0072cc-b1b9-4d2f-8800-4f93d3edac7c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:17.690950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:17.692131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:17.693269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:17.693269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:17.693269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:17.694273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 361/712] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:17.695406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0072cc-b1b9-4d2f-8800-4f93d3edac7c None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:17.725136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:17.750132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449fc77e-eb4b-47db-a0ac-df9b0df5c163 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:17.786165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8235cb0f-8583-4f58-936c-0e6f3c22b80d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:17.786890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8235cb0f-8583-4f58-936c-0e6f3c22b80d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:17.787177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8235cb0f-8583-4f58-936c-0e6f3c22b80d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:17.787476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8235cb0f-8583-4f58-936c-0e6f3c22b80d None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:17.788183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8235cb0f-8583-4f58-936c-0e6f3c22b80d None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:17.788494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8235cb0f-8583-4f58-936c-0e6f3c22b80d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:17.789942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8235cb0f-8583-4f58-936c-0e6f3c22b80d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:17.796433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 352/713] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Sep 13 15:17:17 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:17.797331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8235cb0f-8583-4f58-936c-0e6f3c22b80d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:17.805275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0072cc-b1b9-4d2f-8800-4f93d3edac7c None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:17.832635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:17.844965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ad07f6-8d29-4b30-89d1-20f24b0be3a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:17.846130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ad07f6-8d29-4b30-89d1-20f24b0be3a1 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:17.846622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ad07f6-8d29-4b30-89d1-20f24b0be3a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:17.847095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ad07f6-8d29-4b30-89d1-20f24b0be3a1 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:17.847991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ad07f6-8d29-4b30-89d1-20f24b0be3a1 None admin] RBAC: Adding query filter params (name=network) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:17.848606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ad07f6-8d29-4b30-89d1-20f24b0be3a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:17.850369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ad07f6-8d29-4b30-89d1-20f24b0be3a1 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:17.856219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 362/714] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:17 2022] GET /identity/v3/services?name=network => generated 124 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:17.862810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ad07f6-8d29-4b30-89d1-20f24b0be3a1 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:17.925786 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8235cb0f-8583-4f58-936c-0e6f3c22b80d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:17.969712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3eb2f8-14e4-40c1-8193-a2b089f4711d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:17.970401 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3eb2f8-14e4-40c1-8193-a2b089f4711d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:17.970688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3eb2f8-14e4-40c1-8193-a2b089f4711d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:17.970965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3eb2f8-14e4-40c1-8193-a2b089f4711d None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:17.971622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3eb2f8-14e4-40c1-8193-a2b089f4711d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:17.971926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3eb2f8-14e4-40c1-8193-a2b089f4711d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:17.973987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3eb2f8-14e4-40c1-8193-a2b089f4711d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:17.986184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:17.991821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 353/715] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:17.998137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3eb2f8-14e4-40c1-8193-a2b089f4711d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:18.010748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ad07f6-8d29-4b30-89d1-20f24b0be3a1 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:18.053956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b61006e-6349-4285-b6e5-53d34ec07d23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:18.055128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b61006e-6349-4285-b6e5-53d34ec07d23 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:18.055607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b61006e-6349-4285-b6e5-53d34ec07d23 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:18.056093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b61006e-6349-4285-b6e5-53d34ec07d23 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:18.056940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b61006e-6349-4285-b6e5-53d34ec07d23 None admin] RBAC: Adding query filter params (type=network) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:18.057450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b61006e-6349-4285-b6e5-53d34ec07d23 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:18.059408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b61006e-6349-4285-b6e5-53d34ec07d23 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:18.067414 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 363/716] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:17 2022] GET /identity/v3/services?type=network => generated 124 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:18.193822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3eb2f8-14e4-40c1-8193-a2b089f4711d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:18.252979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04054c24-0933-4f31-aaac-9511709394e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:18.253546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04054c24-0933-4f31-aaac-9511709394e0 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:18.253822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04054c24-0933-4f31-aaac-9511709394e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:18.254115 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04054c24-0933-4f31-aaac-9511709394e0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:18.266063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04054c24-0933-4f31-aaac-9511709394e0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:18.267596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04054c24-0933-4f31-aaac-9511709394e0 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-04054c24-0933-4f31-aaac-9511709394e0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:18.281019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:18.282018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:18.283034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:18.283034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:18.283034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:18.283034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:18.283034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:18.283034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:18.283034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:18.283034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:18.283034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 354/717] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:18.289113 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b61006e-6349-4285-b6e5-53d34ec07d23 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:18.368245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:18.393330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3565ec9-3a87-4d57-b675-763e19b8c282 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:18.393679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3565ec9-3a87-4d57-b675-763e19b8c282 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:18.393881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3565ec9-3a87-4d57-b675-763e19b8c282 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:18.397474 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 355/718] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:18.408650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8377bc9-0415-48fc-9f7c-11f8b8c5de21 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:18.409016 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8377bc9-0415-48fc-9f7c-11f8b8c5de21 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:18.409221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8377bc9-0415-48fc-9f7c-11f8b8c5de21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:18.418725 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b61006e-6349-4285-b6e5-53d34ec07d23 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:18.451211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e8377bc9-0415-48fc-9f7c-11f8b8c5de21 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:18.452519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e8377bc9-0415-48fc-9f7c-11f8b8c5de21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:18.454938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8be4726-77c9-4dd0-921f-ac5d41f77ddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:18.455908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be4726-77c9-4dd0-921f-ac5d41f77ddb None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:18.456333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be4726-77c9-4dd0-921f-ac5d41f77ddb None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:18.456747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8be4726-77c9-4dd0-921f-ac5d41f77ddb None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:18.457592 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8be4726-77c9-4dd0-921f-ac5d41f77ddb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:18.458039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8be4726-77c9-4dd0-921f-ac5d41f77ddb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:18.459694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8be4726-77c9-4dd0-921f-ac5d41f77ddb None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:18.474385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 364/719] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:18.479980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8be4726-77c9-4dd0-921f-ac5d41f77ddb None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:18.567434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:18.591976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 356/720] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:18.667297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8be4726-77c9-4dd0-921f-ac5d41f77ddb None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:18.684405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:18.704975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7acf22-1057-4692-9ab1-81e2477f87ac None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:18.705281 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7acf22-1057-4692-9ab1-81e2477f87ac None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:18.705512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7acf22-1057-4692-9ab1-81e2477f87ac None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:18.707147 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 357/721] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:18.712125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b3c21c-26a4-4455-8640-531a8ca01a65 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:18.712504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b3c21c-26a4-4455-8640-531a8ca01a65 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:18.712776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b3c21c-26a4-4455-8640-531a8ca01a65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:18.713000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe202142-cf50-464a-84b4-3f4fbb475426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:18.713681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe202142-cf50-464a-84b4-3f4fbb475426 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:18.713961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe202142-cf50-464a-84b4-3f4fbb475426 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:18.714262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe202142-cf50-464a-84b4-3f4fbb475426 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:18.714995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe202142-cf50-464a-84b4-3f4fbb475426 None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=35ece665f492449282ab03ad91f0b254) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:18.715307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe202142-cf50-464a-84b4-3f4fbb475426 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:18.717039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe202142-cf50-464a-84b4-3f4fbb475426 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:18.732206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 365/722] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:18 2022] GET /identity/v3/role_assignments?user.id=35ece665f492449282ab03ad91f0b254&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 608 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:18.747312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-45b3c21c-26a4-4455-8640-531a8ca01a65 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:18.748689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-45b3c21c-26a4-4455-8640-531a8ca01a65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:18.848581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 358/723] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 13 15:17:18.902786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe202142-cf50-464a-84b4-3f4fbb475426 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:19.010765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe202142-cf50-464a-84b4-3f4fbb475426 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:19.048905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6b2ff8-682e-4677-9676-dafe64053fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:19.049946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6b2ff8-682e-4677-9676-dafe64053fd5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:19.050448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6b2ff8-682e-4677-9676-dafe64053fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:19.050928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6b2ff8-682e-4677-9676-dafe64053fd5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:19.062033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6b2ff8-682e-4677-9676-dafe64053fd5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-2c6b2ff8-682e-4677-9676-dafe64053fd5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:19.067518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:19.068467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:19.069312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:19.071884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 366/724] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:19.083264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b3c21c-26a4-4455-8640-531a8ca01a65 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:19.210599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b3c21c-26a4-4455-8640-531a8ca01a65 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:19.210804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6b2ff8-682e-4677-9676-dafe64053fd5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:19.239399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:19.256071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2a7dcc-815f-4ee0-a198-61bdc5da24e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:19.256688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2a7dcc-815f-4ee0-a198-61bdc5da24e3 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:19.256961 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2a7dcc-815f-4ee0-a198-61bdc5da24e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:19.257226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2a7dcc-815f-4ee0-a198-61bdc5da24e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:19.258041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2a7dcc-815f-4ee0-a198-61bdc5da24e3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:19.258403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2a7dcc-815f-4ee0-a198-61bdc5da24e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:19.259953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2a7dcc-815f-4ee0-a198-61bdc5da24e3 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:19.264271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:19.270193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 359/725] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:19.278048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2a7dcc-815f-4ee0-a198-61bdc5da24e3 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:19.343508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6b2ff8-682e-4677-9676-dafe64053fd5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:19.383014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbf3382-d822-4687-a1c5-f383b2a35958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:19.384161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbf3382-d822-4687-a1c5-f383b2a35958 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:19.384631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbf3382-d822-4687-a1c5-f383b2a35958 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:19.385088 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbf3382-d822-4687-a1c5-f383b2a35958 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:19.397629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbf3382-d822-4687-a1c5-f383b2a35958 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-0cbf3382-d822-4687-a1c5-f383b2a35958 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:19.402969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:19.404127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:19.404963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:19.407817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 367/726] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:19.414379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbf3382-d822-4687-a1c5-f383b2a35958 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:19.522995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:19.524247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2a7dcc-815f-4ee0-a198-61bdc5da24e3 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:19.530576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbf3382-d822-4687-a1c5-f383b2a35958 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:19.584014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed2d1ed-b844-4773-aac6-8b976b3ddc1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:19.584660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed2d1ed-b844-4773-aac6-8b976b3ddc1c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:19.584933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed2d1ed-b844-4773-aac6-8b976b3ddc1c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:19.585201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed2d1ed-b844-4773-aac6-8b976b3ddc1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:19.587248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3231c65e-0215-4660-98ac-fdc8ea7853ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:19.587892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3231c65e-0215-4660-98ac-fdc8ea7853ce None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:19.588171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3231c65e-0215-4660-98ac-fdc8ea7853ce None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:19.588434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3231c65e-0215-4660-98ac-fdc8ea7853ce None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:19.590296 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3231c65e-0215-4660-98ac-fdc8ea7853ce None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:19.590601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3231c65e-0215-4660-98ac-fdc8ea7853ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:19.592176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3231c65e-0215-4660-98ac-fdc8ea7853ce None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:19.607664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed2d1ed-b844-4773-aac6-8b976b3ddc1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:19.609116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed2d1ed-b844-4773-aac6-8b976b3ddc1c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:19.611174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 368/727] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:19.616795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3231c65e-0215-4660-98ac-fdc8ea7853ce None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-1ed2d1ed-b844-4773-aac6-8b976b3ddc1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:19.645400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:19.646464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:19.647361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:19.647361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:19.647361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:19.647960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 360/728] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 370 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:19.656097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed2d1ed-b844-4773-aac6-8b976b3ddc1c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:19.721235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:19.898196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed2d1ed-b844-4773-aac6-8b976b3ddc1c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:19.937503 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3231c65e-0215-4660-98ac-fdc8ea7853ce None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:19.979284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcfddad-39fc-4b60-9239-394a84766612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:19.979934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcfddad-39fc-4b60-9239-394a84766612 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:19.980210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcfddad-39fc-4b60-9239-394a84766612 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:19.980482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcfddad-39fc-4b60-9239-394a84766612 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:19.987710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af399d-0653-4c5c-9b30-c98903e8e7c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:19.988332 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af399d-0653-4c5c-9b30-c98903e8e7c9 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:19.988952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af399d-0653-4c5c-9b30-c98903e8e7c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:19.989222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af399d-0653-4c5c-9b30-c98903e8e7c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:19.989867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14af399d-0653-4c5c-9b30-c98903e8e7c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:19.990196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14af399d-0653-4c5c-9b30-c98903e8e7c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:19.991602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14af399d-0653-4c5c-9b30-c98903e8e7c9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:20.005939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcfddad-39fc-4b60-9239-394a84766612 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:20.007436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcfddad-39fc-4b60-9239-394a84766612 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:20.019294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 361/729] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 363 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:20.025419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af399d-0653-4c5c-9b30-c98903e8e7c9 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-4bcfddad-39fc-4b60-9239-394a84766612 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:20.032766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:20.033732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:20.034997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:20.034997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:20.034997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:20.034997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 369/730] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 418 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:20.040866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcfddad-39fc-4b60-9239-394a84766612 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:20.066485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:20.149763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcfddad-39fc-4b60-9239-394a84766612 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:20.185207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5621b980-08cc-4223-9bea-93023355fc23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:20.185839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5621b980-08cc-4223-9bea-93023355fc23 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.186129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5621b980-08cc-4223-9bea-93023355fc23 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.186437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5621b980-08cc-4223-9bea-93023355fc23 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.187190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5621b980-08cc-4223-9bea-93023355fc23 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:20.187498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5621b980-08cc-4223-9bea-93023355fc23 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:20.189021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5621b980-08cc-4223-9bea-93023355fc23 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:20.201527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 370/731] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:20.209978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5621b980-08cc-4223-9bea-93023355fc23 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:20.254136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:20.272498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af399d-0653-4c5c-9b30-c98903e8e7c9 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:20.291676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:20.323000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5621b980-08cc-4223-9bea-93023355fc23 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:20.345590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04aa17d2-be0a-4aa0-9afd-dbb1c640b6bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:20.346499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04aa17d2-be0a-4aa0-9afd-dbb1c640b6bf None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.346956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04aa17d2-be0a-4aa0-9afd-dbb1c640b6bf None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.347516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04aa17d2-be0a-4aa0-9afd-dbb1c640b6bf None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.361608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87703301-2698-490e-b9ee-b02fb02f6c42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:20.361608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87703301-2698-490e-b9ee-b02fb02f6c42 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.361608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87703301-2698-490e-b9ee-b02fb02f6c42 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.361608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87703301-2698-490e-b9ee-b02fb02f6c42 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.363932 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04aa17d2-be0a-4aa0-9afd-dbb1c640b6bf None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:20.365561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04aa17d2-be0a-4aa0-9afd-dbb1c640b6bf None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:20.368314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87703301-2698-490e-b9ee-b02fb02f6c42 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:20.369839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87703301-2698-490e-b9ee-b02fb02f6c42 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-87703301-2698-490e-b9ee-b02fb02f6c42 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:20.381969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:20.383235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:20.384412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:20.384412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:20.384412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:20.384412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:20.384412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:20.384412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:20.384412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:20.384412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:20.384412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:20.384412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:20.386686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 371/732] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Sep 13 15:17:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-04aa17d2-be0a-4aa0-9afd-dbb1c640b6bf None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:20.394620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:20.395712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:20.396647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:20.396647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:20.396647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:20.396647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:20.396647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:20.396647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:20.396647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:20.396647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:20.396647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:20.396647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:20.399431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 362/733] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Sep 13 15:17:20 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 374 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:20.401530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87703301-2698-490e-b9ee-b02fb02f6c42 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:20.405400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04aa17d2-be0a-4aa0-9afd-dbb1c640b6bf None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:20.579131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04aa17d2-be0a-4aa0-9afd-dbb1c640b6bf None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:20.581001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87703301-2698-490e-b9ee-b02fb02f6c42 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:20.616845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:20.621168 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48b002c-f484-4ff8-a7e2-3a0dfb256b89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:20.622180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48b002c-f484-4ff8-a7e2-3a0dfb256b89 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.622819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48b002c-f484-4ff8-a7e2-3a0dfb256b89 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.623296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48b002c-f484-4ff8-a7e2-3a0dfb256b89 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.624267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48b002c-f484-4ff8-a7e2-3a0dfb256b89 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:20.624374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec87e857-d4f7-4e69-bc87-93fec2d423fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:20.624880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48b002c-f484-4ff8-a7e2-3a0dfb256b89 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:20.624985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec87e857-d4f7-4e69-bc87-93fec2d423fa None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.625247 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec87e857-d4f7-4e69-bc87-93fec2d423fa None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.625514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec87e857-d4f7-4e69-bc87-93fec2d423fa None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.626277 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec87e857-d4f7-4e69-bc87-93fec2d423fa None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:20.626579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec87e857-d4f7-4e69-bc87-93fec2d423fa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:20.627464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48b002c-f484-4ff8-a7e2-3a0dfb256b89 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:20.627985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec87e857-d4f7-4e69-bc87-93fec2d423fa None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:20.652450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 372/734] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Sep 13 15:17:20 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:20.652450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48746338-4b8a-422a-a4a6-27860e06a6f8 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.652450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48746338-4b8a-422a-a4a6-27860e06a6f8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.652450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48746338-4b8a-422a-a4a6-27860e06a6f8 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.652812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 363/736] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Sep 13 15:17:20 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:20.653422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511c1bf6-a5d0-4c15-93bb-8cc71a09ff34 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.653709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511c1bf6-a5d0-4c15-93bb-8cc71a09ff34 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.654137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511c1bf6-a5d0-4c15-93bb-8cc71a09ff34 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.657011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 373/737] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:20.657128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 364/737] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:20.657215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48746338-4b8a-422a-a4a6-27860e06a6f8 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:20.657290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40963e31-3883-437d-999e-65ed620bb71b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.657290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40963e31-3883-437d-999e-65ed620bb71b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.657545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40963e31-3883-437d-999e-65ed620bb71b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.675342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-40963e31-3883-437d-999e-65ed620bb71b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:20.676527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-40963e31-3883-437d-999e-65ed620bb71b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:20.686753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:20.748680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 365/738] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:20.749393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40963e31-3883-437d-999e-65ed620bb71b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:20.776531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48746338-4b8a-422a-a4a6-27860e06a6f8 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:20.817181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4163ee-f709-46e5-9bc5-255a457945bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:20.818476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4163ee-f709-46e5-9bc5-255a457945bd None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.819011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4163ee-f709-46e5-9bc5-255a457945bd None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.820389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4163ee-f709-46e5-9bc5-255a457945bd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.833038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4163ee-f709-46e5-9bc5-255a457945bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:20.835424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4163ee-f709-46e5-9bc5-255a457945bd None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:20.849990 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40963e31-3883-437d-999e-65ed620bb71b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-6a4163ee-f709-46e5-9bc5-255a457945bd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:20.854080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:20.855034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:20.855988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:20.855988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:20.855988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:20.863773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 374/739] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:20.863910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fd6156-1dfa-4d25-b348-20c627c1e49d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.864506 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fd6156-1dfa-4d25-b348-20c627c1e49d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.865641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fd6156-1dfa-4d25-b348-20c627c1e49d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.889651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-09fd6156-1dfa-4d25-b348-20c627c1e49d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:20.891196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a76d22-3df5-49b2-b682-307f73459ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:20.891579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-09fd6156-1dfa-4d25-b348-20c627c1e49d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:20.891954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a76d22-3df5-49b2-b682-307f73459ccf None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:20.892300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a76d22-3df5-49b2-b682-307f73459ccf None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:20.892684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a76d22-3df5-49b2-b682-307f73459ccf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:20.905936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:20.937252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:20.982081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 375/740] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 13 15:17:20.982081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fd6156-1dfa-4d25-b348-20c627c1e49d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:20.982352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a76d22-3df5-49b2-b682-307f73459ccf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:20.982352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a76d22-3df5-49b2-b682-307f73459ccf None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:21.105428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 366/741] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 13 15:17:21.106462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a76d22-3df5-49b2-b682-307f73459ccf None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:21.111476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fd6156-1dfa-4d25-b348-20c627c1e49d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:21.153106 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:21.156119 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a899e5-d588-4071-b3a7-baa96da62445 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:21.156673 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a899e5-d588-4071-b3a7-baa96da62445 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:21.156945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a899e5-d588-4071-b3a7-baa96da62445 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:21.157249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a899e5-d588-4071-b3a7-baa96da62445 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:21.157892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a899e5-d588-4071-b3a7-baa96da62445 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:21.158371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a899e5-d588-4071-b3a7-baa96da62445 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:21.160911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a899e5-d588-4071-b3a7-baa96da62445 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:21.174899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 376/742] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:21.175863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a899e5-d588-4071-b3a7-baa96da62445 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:21.214709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:21.216599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a76d22-3df5-49b2-b682-307f73459ccf None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:21.254750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37bc9b9-e029-48fa-87f7-9cc57880f0e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:21.255376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37bc9b9-e029-48fa-87f7-9cc57880f0e9 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:21.255657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37bc9b9-e029-48fa-87f7-9cc57880f0e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:21.255933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37bc9b9-e029-48fa-87f7-9cc57880f0e9 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:21.256606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37bc9b9-e029-48fa-87f7-9cc57880f0e9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:21.258080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37bc9b9-e029-48fa-87f7-9cc57880f0e9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:21.281407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 367/743] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:17:21 2022] POST /identity/v3/services => generated 251 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:17:21.282357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37bc9b9-e029-48fa-87f7-9cc57880f0e9 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:21.294276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a899e5-d588-4071-b3a7-baa96da62445 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:21.358616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1c23b4-425c-4656-a7bc-013e17a30229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:21.359281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c23b4-425c-4656-a7bc-013e17a30229 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:21.359558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c23b4-425c-4656-a7bc-013e17a30229 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:21.359826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1c23b4-425c-4656-a7bc-013e17a30229 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:21.372218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1c23b4-425c-4656-a7bc-013e17a30229 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:21.374237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1c23b4-425c-4656-a7bc-013e17a30229 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-bf1c23b4-425c-4656-a7bc-013e17a30229 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:21.396036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:21.397059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:21.397948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:21.397948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:21.397948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:21.397948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 377/744] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:21.399852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1c23b4-425c-4656-a7bc-013e17a30229 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:21.412862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37bc9b9-e029-48fa-87f7-9cc57880f0e9 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:21.414786 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:21.454507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e2cbb2-3130-4f16-ac13-f9020dbec451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:21.455389 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e2cbb2-3130-4f16-ac13-f9020dbec451 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:21.455789 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e2cbb2-3130-4f16-ac13-f9020dbec451 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:21.456153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e2cbb2-3130-4f16-ac13-f9020dbec451 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:21.464214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e2cbb2-3130-4f16-ac13-f9020dbec451 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:21.466048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e2cbb2-3130-4f16-ac13-f9020dbec451 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-27e2cbb2-3130-4f16-ac13-f9020dbec451 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:21.479539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:21.481001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:21.482720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:21.482720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:21.482720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:21.482720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:21.482720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:21.482720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:21.482720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:21.482720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:21.482720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 368/745] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:21.483612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e2cbb2-3130-4f16-ac13-f9020dbec451 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:21.509539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:21.542646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1c23b4-425c-4656-a7bc-013e17a30229 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:21.593228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac43a4b-a4f7-48bb-bb5d-6afe1a0aaf5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:21.594763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac43a4b-a4f7-48bb-bb5d-6afe1a0aaf5f None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:21.595518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac43a4b-a4f7-48bb-bb5d-6afe1a0aaf5f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:21.596017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac43a4b-a4f7-48bb-bb5d-6afe1a0aaf5f None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:21.597167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac43a4b-a4f7-48bb-bb5d-6afe1a0aaf5f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:21.599437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac43a4b-a4f7-48bb-bb5d-6afe1a0aaf5f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:21.606313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e2cbb2-3130-4f16-ac13-f9020dbec451 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:21.622213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 378/746] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:17:21 2022] POST /identity/v3/services => generated 240 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:17:21.623638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac43a4b-a4f7-48bb-bb5d-6afe1a0aaf5f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:21.670680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5198da-a924-4f5b-8555-3ab4de7b088f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:21.670969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5198da-a924-4f5b-8555-3ab4de7b088f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:21.671387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5198da-a924-4f5b-8555-3ab4de7b088f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:21.671724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5198da-a924-4f5b-8555-3ab4de7b088f None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:21.672498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5198da-a924-4f5b-8555-3ab4de7b088f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:21.672860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5198da-a924-4f5b-8555-3ab4de7b088f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:21.674496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5198da-a924-4f5b-8555-3ab4de7b088f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:21.689065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 369/747] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:21.690417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c1a030-8fd1-4ec8-ae7c-4cf1abc54cf4 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:21.690765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c1a030-8fd1-4ec8-ae7c-4cf1abc54cf4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:21.691356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c1a030-8fd1-4ec8-ae7c-4cf1abc54cf4 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:21.692552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 370/748] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:21.697728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c1a030-8fd1-4ec8-ae7c-4cf1abc54cf4 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:21.720767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:21.752701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac43a4b-a4f7-48bb-bb5d-6afe1a0aaf5f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:21.795839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8c3ea5-bf65-42a1-8273-9749da4a56e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:21.796833 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8c3ea5-bf65-42a1-8273-9749da4a56e6 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:21.797299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8c3ea5-bf65-42a1-8273-9749da4a56e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:21.797734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8c3ea5-bf65-42a1-8273-9749da4a56e6 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:21.798646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8c3ea5-bf65-42a1-8273-9749da4a56e6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:21.799225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8c3ea5-bf65-42a1-8273-9749da4a56e6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:21.801091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8c3ea5-bf65-42a1-8273-9749da4a56e6 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:21.810544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:21.822305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 379/749] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:21.824043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a022581a-bbcd-4cfc-a859-93dd5bd5f062 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:21.824890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c1a030-8fd1-4ec8-ae7c-4cf1abc54cf4 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:21.825020 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a022581a-bbcd-4cfc-a859-93dd5bd5f062 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:21.825260 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a022581a-bbcd-4cfc-a859-93dd5bd5f062 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:21.855800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a022581a-bbcd-4cfc-a859-93dd5bd5f062 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:21.857101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a022581a-bbcd-4cfc-a859-93dd5bd5f062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:21.877772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cabb30a-902d-4be4-8a9a-a94d2c16a374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:21.878838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cabb30a-902d-4be4-8a9a-a94d2c16a374 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:21.879600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cabb30a-902d-4be4-8a9a-a94d2c16a374 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:21.880067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cabb30a-902d-4be4-8a9a-a94d2c16a374 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:21.896184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cabb30a-902d-4be4-8a9a-a94d2c16a374 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:21.899247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cabb30a-902d-4be4-8a9a-a94d2c16a374 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-5cabb30a-902d-4be4-8a9a-a94d2c16a374 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:21.916137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:21.917672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:21.918792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:21.918792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:21.918792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:21.918792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:21.918792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:21.918792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:21.918792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:21.918792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:21.926486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 371/750] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:21.928150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cabb30a-902d-4be4-8a9a-a94d2c16a374 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:21.939843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 380/751] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:21 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:21.940807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a022581a-bbcd-4cfc-a859-93dd5bd5f062 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:22.022702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:22.048197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cabb30a-902d-4be4-8a9a-a94d2c16a374 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:22.087935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944368eb-0b14-48a7-86a8-54ddbc85e9cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:22.088613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944368eb-0b14-48a7-86a8-54ddbc85e9cb None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:22.088895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944368eb-0b14-48a7-86a8-54ddbc85e9cb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:22.089170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944368eb-0b14-48a7-86a8-54ddbc85e9cb None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/f027e913f53447a988f74ae4dff84b2c/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:22.092883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944368eb-0b14-48a7-86a8-54ddbc85e9cb None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=f027e913f53447a988f74ae4dff84b2c, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:22.093452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944368eb-0b14-48a7-86a8-54ddbc85e9cb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:22.102380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a022581a-bbcd-4cfc-a859-93dd5bd5f062 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:22.125563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 372/752] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:17:21 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/f027e913f53447a988f74ae4dff84b2c/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:17:22.136466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:22.153997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b134a6fa-4053-4c21-b76d-75b682c44c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:22.155677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b134a6fa-4053-4c21-b76d-75b682c44c09 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:22.156155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b134a6fa-4053-4c21-b76d-75b682c44c09 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:22.156668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b134a6fa-4053-4c21-b76d-75b682c44c09 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:22.157569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b134a6fa-4053-4c21-b76d-75b682c44c09 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:22.158128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b134a6fa-4053-4c21-b76d-75b682c44c09 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:22.159824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b134a6fa-4053-4c21-b76d-75b682c44c09 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:22.179594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 381/753] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:22.191322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944368eb-0b14-48a7-86a8-54ddbc85e9cb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:22.325177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b134a6fa-4053-4c21-b76d-75b682c44c09 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:22.348301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944368eb-0b14-48a7-86a8-54ddbc85e9cb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:22.394475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d36fdf-77e6-4ff7-b98a-2476e6f82c2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:22.395428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d36fdf-77e6-4ff7-b98a-2476e6f82c2c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:22.395873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d36fdf-77e6-4ff7-b98a-2476e6f82c2c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:22.396413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d36fdf-77e6-4ff7-b98a-2476e6f82c2c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:22.397472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d36fdf-77e6-4ff7-b98a-2476e6f82c2c None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=eac26eaa3b1a4c8c849a45ed9ba20301) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:22.398039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d36fdf-77e6-4ff7-b98a-2476e6f82c2c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:22.399774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d36fdf-77e6-4ff7-b98a-2476e6f82c2c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:22.408787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 373/754] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:22 2022] GET /identity/v3/role_assignments?user.id=eac26eaa3b1a4c8c849a45ed9ba20301&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 259 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:22.445697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b134a6fa-4053-4c21-b76d-75b682c44c09 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:22.484428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d726e27-ed92-444b-9249-54557e2ff250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:22.485096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d726e27-ed92-444b-9249-54557e2ff250 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:22.485371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d726e27-ed92-444b-9249-54557e2ff250 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:22.485645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d726e27-ed92-444b-9249-54557e2ff250 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:22.502587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d726e27-ed92-444b-9249-54557e2ff250 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-4d726e27-ed92-444b-9249-54557e2ff250 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:22.508512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:22.509507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:22.510406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:22.513305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 382/755] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:22.520219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d36fdf-77e6-4ff7-b98a-2476e6f82c2c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:22.525622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:22.676319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d36fdf-77e6-4ff7-b98a-2476e6f82c2c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:22.720807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d73086f-d5af-46f2-bb75-77343df11139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:22.721812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d73086f-d5af-46f2-bb75-77343df11139 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:22.722321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d73086f-d5af-46f2-bb75-77343df11139 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:22.722800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d73086f-d5af-46f2-bb75-77343df11139 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:22.723662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d73086f-d5af-46f2-bb75-77343df11139 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:22.724150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d73086f-d5af-46f2-bb75-77343df11139 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:22.725710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d73086f-d5af-46f2-bb75-77343df11139 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:22.737677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 374/756] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:22.743889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d726e27-ed92-444b-9249-54557e2ff250 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:22.829146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:22.895583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d726e27-ed92-444b-9249-54557e2ff250 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:22.949646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02fcbd6-8afc-454a-a669-2030936a06b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:22.950333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02fcbd6-8afc-454a-a669-2030936a06b1 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:22.950822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02fcbd6-8afc-454a-a669-2030936a06b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:22.951541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02fcbd6-8afc-454a-a669-2030936a06b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:22.971380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02fcbd6-8afc-454a-a669-2030936a06b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:22.972772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02fcbd6-8afc-454a-a669-2030936a06b1 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-c02fcbd6-8afc-454a-a669-2030936a06b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:22.988980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:22.990451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:22.992023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:22.992023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:22.992023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:22.992023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 383/757] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:22.997541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d73086f-d5af-46f2-bb75-77343df11139 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:23.102615 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:23.151950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d73086f-d5af-46f2-bb75-77343df11139 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:23.197585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5907bc-d347-4727-af21-c69dfd5e359e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:23.198585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5907bc-d347-4727-af21-c69dfd5e359e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:23.199030 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5907bc-d347-4727-af21-c69dfd5e359e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:23.199311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5907bc-d347-4727-af21-c69dfd5e359e None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:23.200001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5907bc-d347-4727-af21-c69dfd5e359e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:23.200305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5907bc-d347-4727-af21-c69dfd5e359e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:23.202382 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5907bc-d347-4727-af21-c69dfd5e359e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:23.217183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 375/758] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:23.227108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02fcbd6-8afc-454a-a669-2030936a06b1 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:23.383576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02fcbd6-8afc-454a-a669-2030936a06b1 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:23.386426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:23.427964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24730c66-1726-4e9f-b259-ddd9aa10af90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:23.429051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24730c66-1726-4e9f-b259-ddd9aa10af90 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:23.429517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24730c66-1726-4e9f-b259-ddd9aa10af90 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:23.429795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24730c66-1726-4e9f-b259-ddd9aa10af90 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:23.443197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24730c66-1726-4e9f-b259-ddd9aa10af90 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:23.445009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24730c66-1726-4e9f-b259-ddd9aa10af90 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-24730c66-1726-4e9f-b259-ddd9aa10af90 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:23.464496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:23.467189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:23.469345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 384/759] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:23 2022] GET /identity/v3/users/swift => generated 83 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:23.478387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5907bc-d347-4727-af21-c69dfd5e359e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:23.632249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5907bc-d347-4727-af21-c69dfd5e359e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:23.658551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:23.670353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81f57b7-a488-4a7b-887a-29fbf47027c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:23.670981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81f57b7-a488-4a7b-887a-29fbf47027c0 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:23.671258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81f57b7-a488-4a7b-887a-29fbf47027c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:23.671533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81f57b7-a488-4a7b-887a-29fbf47027c0 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:23.672278 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81f57b7-a488-4a7b-887a-29fbf47027c0 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:23.672579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81f57b7-a488-4a7b-887a-29fbf47027c0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:23.674310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81f57b7-a488-4a7b-887a-29fbf47027c0 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:23.698215 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 376/760] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Sep 13 15:17:23 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:23.702810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24730c66-1726-4e9f-b259-ddd9aa10af90 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:23.864702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24730c66-1726-4e9f-b259-ddd9aa10af90 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:23.912257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216a891e-3904-44f4-ad71-b40e49f5abb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:23.913332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216a891e-3904-44f4-ad71-b40e49f5abb1 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:23.913826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216a891e-3904-44f4-ad71-b40e49f5abb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:23.914316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216a891e-3904-44f4-ad71-b40e49f5abb1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:23.958470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:24.016016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216a891e-3904-44f4-ad71-b40e49f5abb1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:24.019042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216a891e-3904-44f4-ad71-b40e49f5abb1 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:24.154879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 385/761] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:23 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:17:24.160627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81f57b7-a488-4a7b-887a-29fbf47027c0 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:24.161111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5962e5ab-de52-4a85-b1fb-74ff22149fa5 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:24.161368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5962e5ab-de52-4a85-b1fb-74ff22149fa5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:24.161642 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5962e5ab-de52-4a85-b1fb-74ff22149fa5 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:24.163237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 386/762] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:24.168990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1e381a-6249-47d4-a7b1-b31a5ed492aa None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:24.169279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1e381a-6249-47d4-a7b1-b31a5ed492aa None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:24.169551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1e381a-6249-47d4-a7b1-b31a5ed492aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:24.203141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-1c1e381a-6249-47d4-a7b1-b31a5ed492aa None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:24.204342 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1c1e381a-6249-47d4-a7b1-b31a5ed492aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:24.295522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81f57b7-a488-4a7b-887a-29fbf47027c0 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:24.328875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:24.334139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc7a227-d57f-4f06-b97c-f9ceb9cd2a70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:24.335473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc7a227-d57f-4f06-b97c-f9ceb9cd2a70 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:24.336117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc7a227-d57f-4f06-b97c-f9ceb9cd2a70 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:24.336702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc7a227-d57f-4f06-b97c-f9ceb9cd2a70 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:24.348191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc7a227-d57f-4f06-b97c-f9ceb9cd2a70 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:24.349913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc7a227-d57f-4f06-b97c-f9ceb9cd2a70 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:24.364409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 387/763] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:24 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-bfc7a227-d57f-4f06-b97c-f9ceb9cd2a70 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:24.367343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:24.368302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:24.369178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:24.369178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:24.369178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:24.371634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 377/764] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:24.377202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1e381a-6249-47d4-a7b1-b31a5ed492aa None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:24.498720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ef8960-33cd-4ff4-be49-63d797c651da None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:24.499657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ef8960-33cd-4ff4-be49-63d797c651da None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:24.500118 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ef8960-33cd-4ff4-be49-63d797c651da None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:24.501621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 378/765] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:24 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:24.507493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29924f22-ea7e-47e8-b994-d83357ee16ca None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:24.507987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29924f22-ea7e-47e8-b994-d83357ee16ca None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:24.508663 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29924f22-ea7e-47e8-b994-d83357ee16ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:24.523655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1e381a-6249-47d4-a7b1-b31a5ed492aa None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:24.545089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-29924f22-ea7e-47e8-b994-d83357ee16ca None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:24.548858 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-29924f22-ea7e-47e8-b994-d83357ee16ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:24.589717 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2530dc5e-4a41-4f8d-a39c-05cf95610a98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:24.590447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2530dc5e-4a41-4f8d-a39c-05cf95610a98 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:24.590724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2530dc5e-4a41-4f8d-a39c-05cf95610a98 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:24.590993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2530dc5e-4a41-4f8d-a39c-05cf95610a98 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:24.591864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2530dc5e-4a41-4f8d-a39c-05cf95610a98 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:24.592337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2530dc5e-4a41-4f8d-a39c-05cf95610a98 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:24.593728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2530dc5e-4a41-4f8d-a39c-05cf95610a98 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:24.608945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:24.609127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 388/766] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:24.615034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2530dc5e-4a41-4f8d-a39c-05cf95610a98 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:24.694561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:24.706187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 379/767] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:24 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:24.707285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29924f22-ea7e-47e8-b994-d83357ee16ca None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:24.751971 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2530dc5e-4a41-4f8d-a39c-05cf95610a98 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:24.789206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236be9d0-ecaf-4f70-97f1-64169e091472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:24.790173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236be9d0-ecaf-4f70-97f1-64169e091472 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:24.790761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236be9d0-ecaf-4f70-97f1-64169e091472 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:24.791207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236be9d0-ecaf-4f70-97f1-64169e091472 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:24.799046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236be9d0-ecaf-4f70-97f1-64169e091472 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:24.800672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236be9d0-ecaf-4f70-97f1-64169e091472 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-236be9d0-ecaf-4f70-97f1-64169e091472 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:24.825281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:24.826885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:24.827973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:24.827973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:24.827973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:24.827973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:24.827973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:24.827973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:24.827973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:24.827973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:24.831375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 389/768] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:24.835765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236be9d0-ecaf-4f70-97f1-64169e091472 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:24.878034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29924f22-ea7e-47e8-b994-d83357ee16ca None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:24.924786 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cedfc4-c74f-4f63-af0b-30f2052da5e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:24.929287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cedfc4-c74f-4f63-af0b-30f2052da5e5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:24.929720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cedfc4-c74f-4f63-af0b-30f2052da5e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:24.930159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cedfc4-c74f-4f63-af0b-30f2052da5e5 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:24.930920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cedfc4-c74f-4f63-af0b-30f2052da5e5 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:24.934313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cedfc4-c74f-4f63-af0b-30f2052da5e5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-a9cedfc4-c74f-4f63-af0b-30f2052da5e5 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:24.950750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:24.951986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:17:24.953034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:24.959913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 380/769] 10.222.0.16 () {56 vars in 1129 bytes} [Tue Sep 13 15:17:24 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:24.965856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cedfc4-c74f-4f63-af0b-30f2052da5e5 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:25.002545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:25.076934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236be9d0-ecaf-4f70-97f1-64169e091472 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:25.089319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:25.119501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cedfc4-c74f-4f63-af0b-30f2052da5e5 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:25.119981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5796e733-7443-4c1f-a937-f7f0de409ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:25.121070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5796e733-7443-4c1f-a937-f7f0de409ef9 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.121496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5796e733-7443-4c1f-a937-f7f0de409ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.122939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5796e733-7443-4c1f-a937-f7f0de409ef9 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.123898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5796e733-7443-4c1f-a937-f7f0de409ef9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:25.124374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5796e733-7443-4c1f-a937-f7f0de409ef9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:25.125973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5796e733-7443-4c1f-a937-f7f0de409ef9 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:25.144982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 390/770] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:25.145832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5796e733-7443-4c1f-a937-f7f0de409ef9 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:25.165836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7225cea-8e69-41c1-b254-3a5b46f69d28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:25.167436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7225cea-8e69-41c1-b254-3a5b46f69d28 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.168311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7225cea-8e69-41c1-b254-3a5b46f69d28 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.169074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7225cea-8e69-41c1-b254-3a5b46f69d28 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.170032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7225cea-8e69-41c1-b254-3a5b46f69d28 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:25.170573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7225cea-8e69-41c1-b254-3a5b46f69d28 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:25.172099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7225cea-8e69-41c1-b254-3a5b46f69d28 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:25.177606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 381/771] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Sep 13 15:17:24 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:25.178466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7225cea-8e69-41c1-b254-3a5b46f69d28 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:25.259989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5796e733-7443-4c1f-a937-f7f0de409ef9 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:25.272787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:25.282719 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7225cea-8e69-41c1-b254-3a5b46f69d28 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:25.297108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c650f3-b2a7-4ff8-bb6d-72b37b9c746b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:25.297887 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c650f3-b2a7-4ff8-bb6d-72b37b9c746b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.298203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c650f3-b2a7-4ff8-bb6d-72b37b9c746b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.298475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c650f3-b2a7-4ff8-bb6d-72b37b9c746b None admin] PATH_INFO: `/v3/services/network` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.299063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c650f3-b2a7-4ff8-bb6d-72b37b9c746b None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:25.300444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c650f3-b2a7-4ff8-bb6d-72b37b9c746b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-22c650f3-b2a7-4ff8-bb6d-72b37b9c746b None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:25.310237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:25.311418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:25.314781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 391/772] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:25 2022] GET /identity/v3/services/network => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:25.315674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884e7ac8-0c6d-48a0-b9aa-09f3f7acbefc None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.315674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884e7ac8-0c6d-48a0-b9aa-09f3f7acbefc None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.315674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884e7ac8-0c6d-48a0-b9aa-09f3f7acbefc None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.315674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 392/773] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:25.315674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884e7ac8-0c6d-48a0-b9aa-09f3f7acbefc None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:25.322464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b118d5-951e-49cd-918d-846c433d3fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:25.323082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b118d5-951e-49cd-918d-846c433d3fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.323356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b118d5-951e-49cd-918d-846c433d3fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.323624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b118d5-951e-49cd-918d-846c433d3fa5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.324348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b118d5-951e-49cd-918d-846c433d3fa5 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=35ece665f492449282ab03ad91f0b254) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:25.324651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b118d5-951e-49cd-918d-846c433d3fa5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:25.326253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b118d5-951e-49cd-918d-846c433d3fa5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:25.331861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 382/774] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:25 2022] GET /identity/v3/role_assignments?user.id=35ece665f492449282ab03ad91f0b254&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=22819414415942d399db534708fe7103 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:25.333065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d486a8-0080-4ac0-b083-a102f1397fdb None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.333351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d486a8-0080-4ac0-b083-a102f1397fdb None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.333620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d486a8-0080-4ac0-b083-a102f1397fdb None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.334859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 383/775] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:25 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:25.339994 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d486a8-0080-4ac0-b083-a102f1397fdb None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:25.356174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:25.439657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884e7ac8-0c6d-48a0-b9aa-09f3f7acbefc None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:25.459214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d486a8-0080-4ac0-b083-a102f1397fdb None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:25.484876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac20baf-4912-4fa5-b21e-b47174936c9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:25.485628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac20baf-4912-4fa5-b21e-b47174936c9d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.485905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac20baf-4912-4fa5-b21e-b47174936c9d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.486214 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac20baf-4912-4fa5-b21e-b47174936c9d None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.486860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac20baf-4912-4fa5-b21e-b47174936c9d None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:25.487164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac20baf-4912-4fa5-b21e-b47174936c9d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:25.489453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac20baf-4912-4fa5-b21e-b47174936c9d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:25.502551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 393/776] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Sep 13 15:17:25 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:25.503960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef3e5f0-fadf-4857-abc7-4ca373d95b1d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.504248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef3e5f0-fadf-4857-abc7-4ca373d95b1d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.504519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef3e5f0-fadf-4857-abc7-4ca373d95b1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.510163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78563968-b8ca-4aa9-bb03-648c80ef270b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:25.510599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78563968-b8ca-4aa9-bb03-648c80ef270b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.510866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78563968-b8ca-4aa9-bb03-648c80ef270b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.511136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78563968-b8ca-4aa9-bb03-648c80ef270b None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.511762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78563968-b8ca-4aa9-bb03-648c80ef270b None admin] RBAC: Adding query filter params (name=network) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:25.512058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78563968-b8ca-4aa9-bb03-648c80ef270b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:25.513498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78563968-b8ca-4aa9-bb03-648c80ef270b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:25.519451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 384/777] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:25 2022] GET /identity/v3/services?name=network => generated 124 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:25.520313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbcd170-553d-4dc9-8152-99a0bf082a3b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.520590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbcd170-553d-4dc9-8152-99a0bf082a3b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.520857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbcd170-553d-4dc9-8152-99a0bf082a3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.529352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-bef3e5f0-fadf-4857-abc7-4ca373d95b1d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:25.530615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-bef3e5f0-fadf-4857-abc7-4ca373d95b1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:25.539612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8bbcd170-553d-4dc9-8152-99a0bf082a3b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:25.540800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8bbcd170-553d-4dc9-8152-99a0bf082a3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:25.556358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:25.602725 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 394/778] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:25 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:25.604724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef3e5f0-fadf-4857-abc7-4ca373d95b1d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:25.616504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 385/779] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:25 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:25.617380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbcd170-553d-4dc9-8152-99a0bf082a3b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:25.650791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:25.752030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef3e5f0-fadf-4857-abc7-4ca373d95b1d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:25.769712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbcd170-553d-4dc9-8152-99a0bf082a3b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:25.805141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c9f6fa-4526-4c61-a635-0d34440bce79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:25.806212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c9f6fa-4526-4c61-a635-0d34440bce79 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.806712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c9f6fa-4526-4c61-a635-0d34440bce79 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.807195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c9f6fa-4526-4c61-a635-0d34440bce79 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.808122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c9f6fa-4526-4c61-a635-0d34440bce79 None admin] RBAC: Adding query filter params (type=network) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:25.808632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c9f6fa-4526-4c61-a635-0d34440bce79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:25.810276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c9f6fa-4526-4c61-a635-0d34440bce79 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:25.815771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 386/780] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:25 2022] GET /identity/v3/services?type=network => generated 350 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:25.816486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07edf1a-b3f5-47ca-9062-e092f9d8d33a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:25.817118 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07edf1a-b3f5-47ca-9062-e092f9d8d33a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:25.817394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07edf1a-b3f5-47ca-9062-e092f9d8d33a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:25.817665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07edf1a-b3f5-47ca-9062-e092f9d8d33a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:25.826154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07edf1a-b3f5-47ca-9062-e092f9d8d33a None admin] RBAC: Adding query filter params (interface=public, service_id=ecfeeb639fcd4343adbfcf57efff9c76, region_id=RegionOne) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:25.826154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07edf1a-b3f5-47ca-9062-e092f9d8d33a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:25.826154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07edf1a-b3f5-47ca-9062-e092f9d8d33a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:25.827595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c9f6fa-4526-4c61-a635-0d34440bce79 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:25.832561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 395/781] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Sep 13 15:17:25 2022] GET /identity/v3/endpoints?service_id=ecfeeb639fcd4343adbfcf57efff9c76&interface=public®ion_id=RegionOne => generated 194 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:25.835278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07edf1a-b3f5-47ca-9062-e092f9d8d33a None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:25.848932 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:25.967859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:25.977759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07edf1a-b3f5-47ca-9062-e092f9d8d33a None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:25.982542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c9f6fa-4526-4c61-a635-0d34440bce79 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:26.014311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed0e2ea-9bc0-4d8e-8858-c523311e59d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:26.014943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed0e2ea-9bc0-4d8e-8858-c523311e59d2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:26.015220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed0e2ea-9bc0-4d8e-8858-c523311e59d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:26.015539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed0e2ea-9bc0-4d8e-8858-c523311e59d2 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:26.016191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed0e2ea-9bc0-4d8e-8858-c523311e59d2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:26.017532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed0e2ea-9bc0-4d8e-8858-c523311e59d2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:26.018047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca7f7a6-b3bb-4fa5-83d5-aa024b0f28c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:26.018694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca7f7a6-b3bb-4fa5-83d5-aa024b0f28c9 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:26.018975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca7f7a6-b3bb-4fa5-83d5-aa024b0f28c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:26.019254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca7f7a6-b3bb-4fa5-83d5-aa024b0f28c9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:26.020006 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca7f7a6-b3bb-4fa5-83d5-aa024b0f28c9 None admin] RBAC: Adding query filter params (interface=public, service_id=7e45f66df6b340828ed6b03adca641e2, region_id=RegionOne) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:26.020261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca7f7a6-b3bb-4fa5-83d5-aa024b0f28c9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:26.021699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca7f7a6-b3bb-4fa5-83d5-aa024b0f28c9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:26.034520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 396/782] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Sep 13 15:17:25 2022] GET /identity/v3/services => generated 772 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:26.034520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed0e2ea-9bc0-4d8e-8858-c523311e59d2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:26.045789 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 387/783] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Sep 13 15:17:25 2022] GET /identity/v3/endpoints?service_id=7e45f66df6b340828ed6b03adca641e2&interface=public®ion_id=RegionOne => generated 194 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:26.046902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca7f7a6-b3bb-4fa5-83d5-aa024b0f28c9 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:26.162214 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed0e2ea-9bc0-4d8e-8858-c523311e59d2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:26.163047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:26.195800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca7f7a6-b3bb-4fa5-83d5-aa024b0f28c9 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:26.206789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4796a93-a181-4b69-8aed-2ebbd83ebe0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:26.208200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4796a93-a181-4b69-8aed-2ebbd83ebe0b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:26.208571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4796a93-a181-4b69-8aed-2ebbd83ebe0b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:26.208957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4796a93-a181-4b69-8aed-2ebbd83ebe0b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:26.223692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4796a93-a181-4b69-8aed-2ebbd83ebe0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-a4796a93-a181-4b69-8aed-2ebbd83ebe0b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:26.230535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:26.231711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:26.232732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:26.233407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 397/784] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:26.234217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4796a93-a181-4b69-8aed-2ebbd83ebe0b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:26.253088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fad668-4afb-42e6-b759-96a6a3447563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:26.254524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fad668-4afb-42e6-b759-96a6a3447563 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:26.255030 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fad668-4afb-42e6-b759-96a6a3447563 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:26.255535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fad668-4afb-42e6-b759-96a6a3447563 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:26.255842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:26.272987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fad668-4afb-42e6-b759-96a6a3447563 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-65fad668-4afb-42e6-b759-96a6a3447563 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:26.280851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:26.281828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:26.282692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:26.285474 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 388/785] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:26.286553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fad668-4afb-42e6-b759-96a6a3447563 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:26.358402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4796a93-a181-4b69-8aed-2ebbd83ebe0b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:26.398641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb4b66-ec5a-41d1-bfd5-9897bd720c6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:26.399420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb4b66-ec5a-41d1-bfd5-9897bd720c6c None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:26.399786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb4b66-ec5a-41d1-bfd5-9897bd720c6c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:26.400143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb4b66-ec5a-41d1-bfd5-9897bd720c6c None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:26.400841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fb4b66-ec5a-41d1-bfd5-9897bd720c6c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:26.403063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fb4b66-ec5a-41d1-bfd5-9897bd720c6c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:26.418141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 398/786] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Sep 13 15:17:26 2022] GET /identity/v3/services => generated 772 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:26.418141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb4b66-ec5a-41d1-bfd5-9897bd720c6c None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:26.418655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fad668-4afb-42e6-b759-96a6a3447563 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:26.488920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec72be2-71a2-4c32-ae17-ae7bbac17b1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:26.489624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec72be2-71a2-4c32-ae17-ae7bbac17b1a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:26.489902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec72be2-71a2-4c32-ae17-ae7bbac17b1a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:26.490194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec72be2-71a2-4c32-ae17-ae7bbac17b1a None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:26.490867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec72be2-71a2-4c32-ae17-ae7bbac17b1a None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:26.491166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec72be2-71a2-4c32-ae17-ae7bbac17b1a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:26.492566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec72be2-71a2-4c32-ae17-ae7bbac17b1a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:26.502508 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:26.502819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 389/787] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:26.508120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec72be2-71a2-4c32-ae17-ae7bbac17b1a None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:26.614584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:26.647460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb4b66-ec5a-41d1-bfd5-9897bd720c6c None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:26.691794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec72be2-71a2-4c32-ae17-ae7bbac17b1a None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:26.695813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99784651-589f-409f-a5ab-d4b5ae0c7c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:26.696666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99784651-589f-409f-a5ab-d4b5ae0c7c53 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:26.697149 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99784651-589f-409f-a5ab-d4b5ae0c7c53 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:26.697565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99784651-589f-409f-a5ab-d4b5ae0c7c53 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:26.698431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99784651-589f-409f-a5ab-d4b5ae0c7c53 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:26.698895 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99784651-589f-409f-a5ab-d4b5ae0c7c53 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:26.701287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99784651-589f-409f-a5ab-d4b5ae0c7c53 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:26.713488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 399/788] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:26.719575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99784651-589f-409f-a5ab-d4b5ae0c7c53 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:26.735150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288ccb1-7ec0-436d-a80f-14a80af500fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:26.735809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288ccb1-7ec0-436d-a80f-14a80af500fb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:26.736089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288ccb1-7ec0-436d-a80f-14a80af500fb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:26.736341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288ccb1-7ec0-436d-a80f-14a80af500fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:26.750588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b288ccb1-7ec0-436d-a80f-14a80af500fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:26.752125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b288ccb1-7ec0-436d-a80f-14a80af500fb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-b288ccb1-7ec0-436d-a80f-14a80af500fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:26.771493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:26.772493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:26.773944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:26.773944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:26.773944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:26.773944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 390/789] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:26.778313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288ccb1-7ec0-436d-a80f-14a80af500fb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:26.837986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99784651-589f-409f-a5ab-d4b5ae0c7c53 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:26.847462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:26.875295 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a67b13-a543-46ba-9dd8-44d78e5bc632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:26.876293 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a67b13-a543-46ba-9dd8-44d78e5bc632 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:26.876783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a67b13-a543-46ba-9dd8-44d78e5bc632 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:26.877257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a67b13-a543-46ba-9dd8-44d78e5bc632 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:26.892770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a67b13-a543-46ba-9dd8-44d78e5bc632 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:26.894650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a67b13-a543-46ba-9dd8-44d78e5bc632 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:26.897355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:26.908817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288ccb1-7ec0-436d-a80f-14a80af500fb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-d1a67b13-a543-46ba-9dd8-44d78e5bc632 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:26.915816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:26.916776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:26.917654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:26.917654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:26.917654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:26.919863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 400/790] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:26.925766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a67b13-a543-46ba-9dd8-44d78e5bc632 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:26.953635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7d194d-da9a-4c63-bfd2-382747fd75cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:26.954646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d194d-da9a-4c63-bfd2-382747fd75cc None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:26.955148 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d194d-da9a-4c63-bfd2-382747fd75cc None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:26.955607 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d194d-da9a-4c63-bfd2-382747fd75cc None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:26.956587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7d194d-da9a-4c63-bfd2-382747fd75cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:26.957076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7d194d-da9a-4c63-bfd2-382747fd75cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:26.958688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7d194d-da9a-4c63-bfd2-382747fd75cc None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:26.971927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 391/791] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:26.978326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7d194d-da9a-4c63-bfd2-382747fd75cc None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:27.104647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a67b13-a543-46ba-9dd8-44d78e5bc632 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:27.146282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc959b95-53c7-4278-98c6-76249b98d8c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:27.146908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc959b95-53c7-4278-98c6-76249b98d8c7 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:27.147184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc959b95-53c7-4278-98c6-76249b98d8c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:27.147466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc959b95-53c7-4278-98c6-76249b98d8c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:27.148110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc959b95-53c7-4278-98c6-76249b98d8c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:27.148415 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc959b95-53c7-4278-98c6-76249b98d8c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:27.149792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc959b95-53c7-4278-98c6-76249b98d8c7 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:27.172063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7d194d-da9a-4c63-bfd2-382747fd75cc None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:27.176516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 401/792] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:27.183173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc959b95-53c7-4278-98c6-76249b98d8c7 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:27.189742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:27.212114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb30eda-6728-4d54-a09a-77f2e78a85ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:27.212750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb30eda-6728-4d54-a09a-77f2e78a85ed None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:27.213010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb30eda-6728-4d54-a09a-77f2e78a85ed None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:27.213285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb30eda-6728-4d54-a09a-77f2e78a85ed None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:27.221092 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb30eda-6728-4d54-a09a-77f2e78a85ed None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:27.222562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb30eda-6728-4d54-a09a-77f2e78a85ed None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-ccb30eda-6728-4d54-a09a-77f2e78a85ed None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:27.234945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:27.235931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:27.237862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 392/793] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Sep 13 15:17:26 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:27.242137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb30eda-6728-4d54-a09a-77f2e78a85ed None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:27.270133 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:27.344945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc959b95-53c7-4278-98c6-76249b98d8c7 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:27.391811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96b9c44-1050-4dd9-8855-565b6e1e1e81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:27.392458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96b9c44-1050-4dd9-8855-565b6e1e1e81 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:27.392736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96b9c44-1050-4dd9-8855-565b6e1e1e81 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:27.393006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96b9c44-1050-4dd9-8855-565b6e1e1e81 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:27.400448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96b9c44-1050-4dd9-8855-565b6e1e1e81 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:27.401818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96b9c44-1050-4dd9-8855-565b6e1e1e81 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-f96b9c44-1050-4dd9-8855-565b6e1e1e81 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:27.413595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:27.415334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:27.417110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 402/794] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Sep 13 15:17:27 2022] GET /identity/v3/users/nova => generated 82 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:27.425102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96b9c44-1050-4dd9-8855-565b6e1e1e81 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:27.454462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb30eda-6728-4d54-a09a-77f2e78a85ed None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:27.527314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d73a4cb-a1cd-4b45-af1b-3390eb2a1de3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:27.528358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d73a4cb-a1cd-4b45-af1b-3390eb2a1de3 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:27.529304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d73a4cb-a1cd-4b45-af1b-3390eb2a1de3 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:27.530763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d73a4cb-a1cd-4b45-af1b-3390eb2a1de3 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:27.531983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d73a4cb-a1cd-4b45-af1b-3390eb2a1de3 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:27.535586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d73a4cb-a1cd-4b45-af1b-3390eb2a1de3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:27.537421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d73a4cb-a1cd-4b45-af1b-3390eb2a1de3 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:27.574587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 393/795] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Sep 13 15:17:27 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:27.578896 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d73a4cb-a1cd-4b45-af1b-3390eb2a1de3 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:27.630613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96b9c44-1050-4dd9-8855-565b6e1e1e81 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:27.641920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:27.651796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:27.676233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabd3ed8-a1bc-40d4-895b-00b1e7472d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:27.677882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabd3ed8-a1bc-40d4-895b-00b1e7472d2a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:27.678377 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabd3ed8-a1bc-40d4-895b-00b1e7472d2a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:27.678959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabd3ed8-a1bc-40d4-895b-00b1e7472d2a None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:27.679967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabd3ed8-a1bc-40d4-895b-00b1e7472d2a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:27.680464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabd3ed8-a1bc-40d4-895b-00b1e7472d2a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:27.682089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabd3ed8-a1bc-40d4-895b-00b1e7472d2a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:27.707257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 403/796] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Sep 13 15:17:27 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:27.711035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce7aeac-f6d3-44fa-924a-f9ccd360b268 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:27.711719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce7aeac-f6d3-44fa-924a-f9ccd360b268 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:27.712209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce7aeac-f6d3-44fa-924a-f9ccd360b268 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:27.714487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 404/797] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:27 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:27.720440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce7aeac-f6d3-44fa-924a-f9ccd360b268 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:27.726828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d73a4cb-a1cd-4b45-af1b-3390eb2a1de3 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:27.780444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f120da3-0912-42c8-a9e2-bf9652b1cb2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:27.781076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f120da3-0912-42c8-a9e2-bf9652b1cb2c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:27.781347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f120da3-0912-42c8-a9e2-bf9652b1cb2c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:27.781575 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f120da3-0912-42c8-a9e2-bf9652b1cb2c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:27.793738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f120da3-0912-42c8-a9e2-bf9652b1cb2c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:27.796401 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f120da3-0912-42c8-a9e2-bf9652b1cb2c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-9f120da3-0912-42c8-a9e2-bf9652b1cb2c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:27.821869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:27.822890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:27.823773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:27.823773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:27.823773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:27.824365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 394/798] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:27.826074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be03a654-6275-4b21-814d-acf52dbd7834 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:27.826285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be03a654-6275-4b21-814d-acf52dbd7834 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:27.826630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be03a654-6275-4b21-814d-acf52dbd7834 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:27.849126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-be03a654-6275-4b21-814d-acf52dbd7834 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:27.850429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-be03a654-6275-4b21-814d-acf52dbd7834 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:27.855276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce7aeac-f6d3-44fa-924a-f9ccd360b268 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:27.896629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85f4548-7113-4906-af07-116bb823b5cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:27.897753 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85f4548-7113-4906-af07-116bb823b5cb None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:27.898349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85f4548-7113-4906-af07-116bb823b5cb None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:27.898821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85f4548-7113-4906-af07-116bb823b5cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:27.933511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 395/799] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:27 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:27.933511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be03a654-6275-4b21-814d-acf52dbd7834 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:27.960422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:27.963871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:28.020173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85f4548-7113-4906-af07-116bb823b5cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:28.021998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85f4548-7113-4906-af07-116bb823b5cb None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:28.139226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be03a654-6275-4b21-814d-acf52dbd7834 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:28.143286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 405/800] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:27 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:17:28.152369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85f4548-7113-4906-af07-116bb823b5cb None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:28.216174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5b7cb2-5f21-4881-8883-289a83a74cfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:28.216702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5b7cb2-5f21-4881-8883-289a83a74cfe None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:28.216984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5b7cb2-5f21-4881-8883-289a83a74cfe None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:28.217254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5b7cb2-5f21-4881-8883-289a83a74cfe None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:28.217905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5b7cb2-5f21-4881-8883-289a83a74cfe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:28.218232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5b7cb2-5f21-4881-8883-289a83a74cfe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:28.229247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5b7cb2-5f21-4881-8883-289a83a74cfe None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:28.247863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 396/801] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:28.255491 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5b7cb2-5f21-4881-8883-289a83a74cfe None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:28.313579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:28.338310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:28.353096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85f4548-7113-4906-af07-116bb823b5cb None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:28.371485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5b7cb2-5f21-4881-8883-289a83a74cfe None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:28.394804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c3957-5288-484b-b122-273319cebe80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:28.395748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c3957-5288-484b-b122-273319cebe80 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:28.396206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c3957-5288-484b-b122-273319cebe80 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:28.396621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808c3957-5288-484b-b122-273319cebe80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:28.410746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550eb1dd-e9aa-4937-902f-5fc13a803d54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:28.411339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808c3957-5288-484b-b122-273319cebe80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:28.411917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550eb1dd-e9aa-4937-902f-5fc13a803d54 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:28.412328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550eb1dd-e9aa-4937-902f-5fc13a803d54 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:28.412691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550eb1dd-e9aa-4937-902f-5fc13a803d54 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:28.413469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808c3957-5288-484b-b122-273319cebe80 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:28.420006 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550eb1dd-e9aa-4937-902f-5fc13a803d54 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:28.421643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550eb1dd-e9aa-4937-902f-5fc13a803d54 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-808c3957-5288-484b-b122-273319cebe80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:28.429711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:28.430906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:28.431881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:28.431881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:28.431881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:28.433791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 406/802] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-550eb1dd-e9aa-4937-902f-5fc13a803d54 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:28.434609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:28.435527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:28.436407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:28.436407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:28.436407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:28.436407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:28.436407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:28.436407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:28.436407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:28.436407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:28.436407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 397/803] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:28.437593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c3957-5288-484b-b122-273319cebe80 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:28.445521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550eb1dd-e9aa-4937-902f-5fc13a803d54 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:28.554210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808c3957-5288-484b-b122-273319cebe80 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:28.564036 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550eb1dd-e9aa-4937-902f-5fc13a803d54 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:28.579332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:28.592675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740fdf2e-5be9-41b4-9ab2-a6e5cf6d02f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:28.593318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740fdf2e-5be9-41b4-9ab2-a6e5cf6d02f3 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:28.593592 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740fdf2e-5be9-41b4-9ab2-a6e5cf6d02f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:28.593857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740fdf2e-5be9-41b4-9ab2-a6e5cf6d02f3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:28.595750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:28.601832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080523c2-8167-4dc8-9274-67405091939c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:28.602477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080523c2-8167-4dc8-9274-67405091939c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:28.602759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080523c2-8167-4dc8-9274-67405091939c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:28.603031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080523c2-8167-4dc8-9274-67405091939c None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:28.603726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080523c2-8167-4dc8-9274-67405091939c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:28.604036 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080523c2-8167-4dc8-9274-67405091939c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:28.604551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740fdf2e-5be9-41b4-9ab2-a6e5cf6d02f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:28.605441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080523c2-8167-4dc8-9274-67405091939c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-740fdf2e-5be9-41b4-9ab2-a6e5cf6d02f3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:28.609917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:28.610867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:28.611711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:28.616450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 407/804] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:28.617301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740fdf2e-5be9-41b4-9ab2-a6e5cf6d02f3 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:28.623745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 398/805] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:28.623745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a459511-cd4b-4698-9ad2-244e8a0a7c92 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:28.623745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a459511-cd4b-4698-9ad2-244e8a0a7c92 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:28.623745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a459511-cd4b-4698-9ad2-244e8a0a7c92 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:28.623745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 399/806] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:28 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:28.623745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a459511-cd4b-4698-9ad2-244e8a0a7c92 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:28.730483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a459511-cd4b-4698-9ad2-244e8a0a7c92 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:28.733092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740fdf2e-5be9-41b4-9ab2-a6e5cf6d02f3 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:28.766922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be5b52d-eed8-4da5-ad59-b0115a6acf08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:28.767810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be5b52d-eed8-4da5-ad59-b0115a6acf08 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:28.768229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be5b52d-eed8-4da5-ad59-b0115a6acf08 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:28.768630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be5b52d-eed8-4da5-ad59-b0115a6acf08 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:28.769114 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce59a87e-9c56-489f-bbe4-2e1fbc20fbd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:28.769594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be5b52d-eed8-4da5-ad59-b0115a6acf08 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:28.769730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce59a87e-9c56-489f-bbe4-2e1fbc20fbd7 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:28.770015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce59a87e-9c56-489f-bbe4-2e1fbc20fbd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:28.770325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce59a87e-9c56-489f-bbe4-2e1fbc20fbd7 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:28.770442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be5b52d-eed8-4da5-ad59-b0115a6acf08 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:28.771017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce59a87e-9c56-489f-bbe4-2e1fbc20fbd7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:28.771325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce59a87e-9c56-489f-bbe4-2e1fbc20fbd7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:28.772206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be5b52d-eed8-4da5-ad59-b0115a6acf08 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:28.772749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce59a87e-9c56-489f-bbe4-2e1fbc20fbd7 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:28.785522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 408/807] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:28.785707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce59a87e-9c56-489f-bbe4-2e1fbc20fbd7 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:28.786356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 400/808] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:28.787823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ce0dd-fe20-4858-80e7-c64c83a82503 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:28.788104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ce0dd-fe20-4858-80e7-c64c83a82503 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:28.788371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01ce0dd-fe20-4858-80e7-c64c83a82503 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:28.807704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-f01ce0dd-fe20-4858-80e7-c64c83a82503 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:28.808831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f01ce0dd-fe20-4858-80e7-c64c83a82503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:28.839028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:28.861523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:28.879909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 401/809] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:28.891402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01ce0dd-fe20-4858-80e7-c64c83a82503 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:28.902148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce59a87e-9c56-489f-bbe4-2e1fbc20fbd7 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:28.942725 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5688a0-6a0c-4448-a6ae-5e26117fed18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:28.943368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5688a0-6a0c-4448-a6ae-5e26117fed18 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:28.943647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5688a0-6a0c-4448-a6ae-5e26117fed18 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:28.943918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5688a0-6a0c-4448-a6ae-5e26117fed18 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:28.951121 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5688a0-6a0c-4448-a6ae-5e26117fed18 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:28.952530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5688a0-6a0c-4448-a6ae-5e26117fed18 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-de5688a0-6a0c-4448-a6ae-5e26117fed18 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:28.963957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:28.964853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:28.965719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:28.965719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:28.965719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:28.965719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:28.965719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:28.965719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:28.965719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:28.965719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:28.970072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 409/810] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:28.970952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5688a0-6a0c-4448-a6ae-5e26117fed18 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:29.016780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01ce0dd-fe20-4858-80e7-c64c83a82503 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:29.049880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c854506e-ac78-474b-8ff9-50616ea29d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:29.050538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c854506e-ac78-474b-8ff9-50616ea29d96 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.050812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c854506e-ac78-474b-8ff9-50616ea29d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.051070 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c854506e-ac78-474b-8ff9-50616ea29d96 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.061529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c854506e-ac78-474b-8ff9-50616ea29d96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:29.062938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c854506e-ac78-474b-8ff9-50616ea29d96 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:29.073814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5688a0-6a0c-4448-a6ae-5e26117fed18 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:29.077014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-c854506e-ac78-474b-8ff9-50616ea29d96 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:29.078007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:29.078982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:29.079848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:29.079848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:29.079848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:29.079848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 402/811] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:29.080576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c854506e-ac78-474b-8ff9-50616ea29d96 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:29.114712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356cec90-a01c-4e70-ba4c-1d61aaa677ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:29.115631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356cec90-a01c-4e70-ba4c-1d61aaa677ac None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.115905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356cec90-a01c-4e70-ba4c-1d61aaa677ac None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.116176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356cec90-a01c-4e70-ba4c-1d61aaa677ac None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/eac26eaa3b1a4c8c849a45ed9ba20301/roles/da0ea75123ba496fb410e87b8df992ba` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.117651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:29.132589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356cec90-a01c-4e70-ba4c-1d61aaa677ac None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=eac26eaa3b1a4c8c849a45ed9ba20301, role_id=da0ea75123ba496fb410e87b8df992ba)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:29.134015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356cec90-a01c-4e70-ba4c-1d61aaa677ac None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:29.155206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 410/812] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:17:28 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/eac26eaa3b1a4c8c849a45ed9ba20301/roles/da0ea75123ba496fb410e87b8df992ba => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:17:29.157086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb237d0-0c20-4cb9-87b6-83408a717fe5 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.157554 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb237d0-0c20-4cb9-87b6-83408a717fe5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.158000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb237d0-0c20-4cb9-87b6-83408a717fe5 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.159549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 411/813] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:29.160241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb237d0-0c20-4cb9-87b6-83408a717fe5 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:29.194673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c854506e-ac78-474b-8ff9-50616ea29d96 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:29.234511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845ef85f-2c80-4fa4-8c7b-350d829778f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:29.235208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ef85f-2c80-4fa4-8c7b-350d829778f8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.235404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ef85f-2c80-4fa4-8c7b-350d829778f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.235675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ef85f-2c80-4fa4-8c7b-350d829778f8 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.236407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845ef85f-2c80-4fa4-8c7b-350d829778f8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:29.236683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845ef85f-2c80-4fa4-8c7b-350d829778f8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:29.238195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845ef85f-2c80-4fa4-8c7b-350d829778f8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:29.251827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 403/814] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:29.252726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845ef85f-2c80-4fa4-8c7b-350d829778f8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:29.296604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb237d0-0c20-4cb9-87b6-83408a717fe5 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:29.340620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b927e2dc-a6ba-41dc-a26d-a40c565aa10b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:29.341226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b927e2dc-a6ba-41dc-a26d-a40c565aa10b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.341489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b927e2dc-a6ba-41dc-a26d-a40c565aa10b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.341761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b927e2dc-a6ba-41dc-a26d-a40c565aa10b None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.342472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b927e2dc-a6ba-41dc-a26d-a40c565aa10b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:29.343272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b927e2dc-a6ba-41dc-a26d-a40c565aa10b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:29.344680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b927e2dc-a6ba-41dc-a26d-a40c565aa10b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:29.356621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 412/815] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:29.357789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a787722b-18d8-4bc0-948c-fc455c574244 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.358179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a787722b-18d8-4bc0-948c-fc455c574244 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.358411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a787722b-18d8-4bc0-948c-fc455c574244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.362483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:29.370928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845ef85f-2c80-4fa4-8c7b-350d829778f8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:29.377141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a787722b-18d8-4bc0-948c-fc455c574244 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:29.378349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a787722b-18d8-4bc0-948c-fc455c574244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:29.388553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:29.407347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34444948-1a2d-4b15-9fe5-87ddb5a7cd07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:29.407991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34444948-1a2d-4b15-9fe5-87ddb5a7cd07 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.408268 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34444948-1a2d-4b15-9fe5-87ddb5a7cd07 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.408808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34444948-1a2d-4b15-9fe5-87ddb5a7cd07 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.409128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34444948-1a2d-4b15-9fe5-87ddb5a7cd07 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:29.410531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34444948-1a2d-4b15-9fe5-87ddb5a7cd07 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-34444948-1a2d-4b15-9fe5-87ddb5a7cd07 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:29.419490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:29.420494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:29.421330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 404/816] 10.222.0.16 () {56 vars in 1129 bytes} [Tue Sep 13 15:17:29 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:29.422708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34444948-1a2d-4b15-9fe5-87ddb5a7cd07 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:29.456148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 413/817] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:29 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:29.457082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a787722b-18d8-4bc0-948c-fc455c574244 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:29.538656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34444948-1a2d-4b15-9fe5-87ddb5a7cd07 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:29.577747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a787722b-18d8-4bc0-948c-fc455c574244 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:29.580246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b36dc7b-eb69-452b-aebb-9c297ec78610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:29.581284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b36dc7b-eb69-452b-aebb-9c297ec78610 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.581284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b36dc7b-eb69-452b-aebb-9c297ec78610 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.581482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b36dc7b-eb69-452b-aebb-9c297ec78610 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.582129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b36dc7b-eb69-452b-aebb-9c297ec78610 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=f027e913f53447a988f74ae4dff84b2c) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:29.582435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b36dc7b-eb69-452b-aebb-9c297ec78610 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:29.583877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b36dc7b-eb69-452b-aebb-9c297ec78610 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:29.597761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 405/818] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:29 2022] GET /identity/v3/role_assignments?user.id=f027e913f53447a988f74ae4dff84b2c&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=22819414415942d399db534708fe7103 => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:29.598865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b36dc7b-eb69-452b-aebb-9c297ec78610 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:29.619209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b43bd0-d4c5-4888-b410-24350d5a7597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:29.620332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b43bd0-d4c5-4888-b410-24350d5a7597 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.620810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b43bd0-d4c5-4888-b410-24350d5a7597 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.621266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b43bd0-d4c5-4888-b410-24350d5a7597 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.633456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b43bd0-d4c5-4888-b410-24350d5a7597 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:29.635608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b43bd0-d4c5-4888-b410-24350d5a7597 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-64b43bd0-d4c5-4888-b410-24350d5a7597 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:29.653122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:29.654267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:29.655288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:29.655288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:29.655288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:29.655288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:29.655288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:29.655288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:29.655288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:29.655288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:29.655288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:29.655288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:29.658260 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 414/819] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:29 2022] GET /identity/v3/users/swift => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:29.663567 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:29.664088 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b43bd0-d4c5-4888-b410-24350d5a7597 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:29.671648 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:29.792590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b36dc7b-eb69-452b-aebb-9c297ec78610 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:29.823499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b43bd0-d4c5-4888-b410-24350d5a7597 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:29.830013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36357b-db19-458e-a72b-a640fbd3f89a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:29.830662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36357b-db19-458e-a72b-a640fbd3f89a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.830935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36357b-db19-458e-a72b-a640fbd3f89a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.831202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36357b-db19-458e-a72b-a640fbd3f89a None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.831829 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed36357b-db19-458e-a72b-a640fbd3f89a None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:29.832126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed36357b-db19-458e-a72b-a640fbd3f89a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:29.833498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed36357b-db19-458e-a72b-a640fbd3f89a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:29.838512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 406/820] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Sep 13 15:17:29 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:29.839322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36357b-db19-458e-a72b-a640fbd3f89a None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:29.866680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644d6752-f0ce-4cc8-83af-8ac70af24860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:29.867230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644d6752-f0ce-4cc8-83af-8ac70af24860 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:29.867508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644d6752-f0ce-4cc8-83af-8ac70af24860 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:29.867778 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644d6752-f0ce-4cc8-83af-8ac70af24860 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:29.868511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644d6752-f0ce-4cc8-83af-8ac70af24860 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:29.868817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644d6752-f0ce-4cc8-83af-8ac70af24860 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:29.870458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644d6752-f0ce-4cc8-83af-8ac70af24860 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:29.898836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 415/821] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Sep 13 15:17:29 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:29.899840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644d6752-f0ce-4cc8-83af-8ac70af24860 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:29.953056 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:29.970275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36357b-db19-458e-a72b-a640fbd3f89a None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:29.977784 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:30.013045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e856dc6-0ef4-416b-b206-7777fde23f98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:30.014042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e856dc6-0ef4-416b-b206-7777fde23f98 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:30.014664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e856dc6-0ef4-416b-b206-7777fde23f98 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:30.015178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e856dc6-0ef4-416b-b206-7777fde23f98 None admin] PATH_INFO: `/v3/services/network` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:30.015964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e856dc6-0ef4-416b-b206-7777fde23f98 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:30.017638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e856dc6-0ef4-416b-b206-7777fde23f98 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:30.023753 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644d6752-f0ce-4cc8-83af-8ac70af24860 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-6e856dc6-0ef4-416b-b206-7777fde23f98 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:30.027440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:30.028537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 13 15:17:30.029599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:30.032192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 407/822] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:29 2022] GET /identity/v3/services/network => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:30.033199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e856dc6-0ef4-416b-b206-7777fde23f98 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:30.059019 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6038b38a-dba0-44bf-aa99-f1ef9740dc40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:30.059662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6038b38a-dba0-44bf-aa99-f1ef9740dc40 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:30.060155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6038b38a-dba0-44bf-aa99-f1ef9740dc40 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:30.060467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6038b38a-dba0-44bf-aa99-f1ef9740dc40 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:30.061154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6038b38a-dba0-44bf-aa99-f1ef9740dc40 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:30.061688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6038b38a-dba0-44bf-aa99-f1ef9740dc40 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:30.062978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6038b38a-dba0-44bf-aa99-f1ef9740dc40 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:30.072975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 416/823] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Sep 13 15:17:29 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:30.073962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6038b38a-dba0-44bf-aa99-f1ef9740dc40 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:30.143291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e856dc6-0ef4-416b-b206-7777fde23f98 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:30.178913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a84995-9253-4c47-afe3-a9a7f118c6d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:30.179167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6038b38a-dba0-44bf-aa99-f1ef9740dc40 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:30.180052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a84995-9253-4c47-afe3-a9a7f118c6d1 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:30.180619 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a84995-9253-4c47-afe3-a9a7f118c6d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:30.181079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a84995-9253-4c47-afe3-a9a7f118c6d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:30.199030 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a84995-9253-4c47-afe3-a9a7f118c6d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:30.200602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a84995-9253-4c47-afe3-a9a7f118c6d1 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:30.207996 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-d9a84995-9253-4c47-afe3-a9a7f118c6d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:30.217864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:30.218877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:30.219822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:30.219822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:30.219822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:30.221565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 408/824] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:30.222610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a84995-9253-4c47-afe3-a9a7f118c6d1 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:30.223424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a12bf3-c815-42b9-8e30-1bb9a62ab788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:30.224134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a12bf3-c815-42b9-8e30-1bb9a62ab788 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:30.224539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a12bf3-c815-42b9-8e30-1bb9a62ab788 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:30.224896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a12bf3-c815-42b9-8e30-1bb9a62ab788 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:30.225620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a12bf3-c815-42b9-8e30-1bb9a62ab788 None admin] RBAC: Adding query filter params (name=network) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:30.226220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a12bf3-c815-42b9-8e30-1bb9a62ab788 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:30.227960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:30.228397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a12bf3-c815-42b9-8e30-1bb9a62ab788 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:30.235409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 417/825] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:30 2022] GET /identity/v3/services?name=network => generated 124 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:30.236364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a12bf3-c815-42b9-8e30-1bb9a62ab788 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:30.333007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a84995-9253-4c47-afe3-a9a7f118c6d1 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:30.347218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a12bf3-c815-42b9-8e30-1bb9a62ab788 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:30.369729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1ba981-9c49-4373-b938-5f055ec00d40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:30.370597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1ba981-9c49-4373-b938-5f055ec00d40 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:30.371084 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1ba981-9c49-4373-b938-5f055ec00d40 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:30.371523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1ba981-9c49-4373-b938-5f055ec00d40 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:30.372512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1ba981-9c49-4373-b938-5f055ec00d40 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:30.374174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1ba981-9c49-4373-b938-5f055ec00d40 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:30.384381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1546a3-e51d-44e3-9f7c-d635dc953840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:30.385251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1546a3-e51d-44e3-9f7c-d635dc953840 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:30.385595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1546a3-e51d-44e3-9f7c-d635dc953840 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:30.386419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1546a3-e51d-44e3-9f7c-d635dc953840 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:30.386855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1546a3-e51d-44e3-9f7c-d635dc953840 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:30.387224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1546a3-e51d-44e3-9f7c-d635dc953840 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:30.389085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1546a3-e51d-44e3-9f7c-d635dc953840 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:30.401440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 409/826] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:17:30 2022] POST /identity/v3/endpoints => generated 346 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:17:30.402618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1ba981-9c49-4373-b938-5f055ec00d40 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:30.411878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 418/827] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:30.417504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1546a3-e51d-44e3-9f7c-d635dc953840 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:30.471377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:30.520587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:30.552976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1ba981-9c49-4373-b938-5f055ec00d40 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:30.603140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1546a3-e51d-44e3-9f7c-d635dc953840 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:30.629434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf99fc83-6a1b-467b-8bd4-51f09e2fce35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:30.629434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf99fc83-6a1b-467b-8bd4-51f09e2fce35 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:30.629434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf99fc83-6a1b-467b-8bd4-51f09e2fce35 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:30.629434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf99fc83-6a1b-467b-8bd4-51f09e2fce35 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:30.631489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf99fc83-6a1b-467b-8bd4-51f09e2fce35 None admin] RBAC: Adding query filter params (type=network) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:30.631838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf99fc83-6a1b-467b-8bd4-51f09e2fce35 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:30.633261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf99fc83-6a1b-467b-8bd4-51f09e2fce35 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:30.648561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 410/828] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:30 2022] GET /identity/v3/services?type=network => generated 350 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:30.655600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf99fc83-6a1b-467b-8bd4-51f09e2fce35 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:30.686396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451f7eaa-30fc-42bd-ae06-b6da33971be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:30.687194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451f7eaa-30fc-42bd-ae06-b6da33971be2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:30.687476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451f7eaa-30fc-42bd-ae06-b6da33971be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:30.687742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451f7eaa-30fc-42bd-ae06-b6da33971be2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:30.695226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451f7eaa-30fc-42bd-ae06-b6da33971be2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:30.696677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451f7eaa-30fc-42bd-ae06-b6da33971be2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-451f7eaa-30fc-42bd-ae06-b6da33971be2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:30.710613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:30.711695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:30.712840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:30.712840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:30.712840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:30.712840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:30.712840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:30.712840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:30.712840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:30.712840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:30.716810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 419/829] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:30.718182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451f7eaa-30fc-42bd-ae06-b6da33971be2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:30.785448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf99fc83-6a1b-467b-8bd4-51f09e2fce35 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:30.815014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:30.835149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6babe02e-acda-4786-9497-cc2f8d637a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:30.835771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6babe02e-acda-4786-9497-cc2f8d637a3b None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:30.836193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6babe02e-acda-4786-9497-cc2f8d637a3b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:30.836464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6babe02e-acda-4786-9497-cc2f8d637a3b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:30.837459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6babe02e-acda-4786-9497-cc2f8d637a3b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:30.839228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6babe02e-acda-4786-9497-cc2f8d637a3b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:30.839859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451f7eaa-30fc-42bd-ae06-b6da33971be2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:30.853112 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:30.880042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 411/830] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:17:30 2022] POST /identity/v3/endpoints => generated 337 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:17:30.907606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0d1141-2e5b-4c9d-a890-4c6de5b35820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:30.908207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0d1141-2e5b-4c9d-a890-4c6de5b35820 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:30.908479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0d1141-2e5b-4c9d-a890-4c6de5b35820 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:30.908745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0d1141-2e5b-4c9d-a890-4c6de5b35820 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:30.909443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0d1141-2e5b-4c9d-a890-4c6de5b35820 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:30.909744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0d1141-2e5b-4c9d-a890-4c6de5b35820 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:30.911254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0d1141-2e5b-4c9d-a890-4c6de5b35820 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:30.929431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 420/831] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:30.931241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6babe02e-acda-4786-9497-cc2f8d637a3b None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:31.063922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6babe02e-acda-4786-9497-cc2f8d637a3b None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:31.102454 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46447e1c-313e-4437-9e36-84df3006f097 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:31.103561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46447e1c-313e-4437-9e36-84df3006f097 None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:31.104175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46447e1c-313e-4437-9e36-84df3006f097 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:31.104644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46447e1c-313e-4437-9e36-84df3006f097 None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/35ece665f492449282ab03ad91f0b254/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:31.107551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46447e1c-313e-4437-9e36-84df3006f097 None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=35ece665f492449282ab03ad91f0b254, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:31.109233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46447e1c-313e-4437-9e36-84df3006f097 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:31.131292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 412/832] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:17:30 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/35ece665f492449282ab03ad91f0b254/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:17:31.563325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378b875f-2a4e-4334-be54-06f783f9e6b9 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:31.564141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378b875f-2a4e-4334-be54-06f783f9e6b9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:31.564679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378b875f-2a4e-4334-be54-06f783f9e6b9 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:31.566504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 421/833] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:31.572209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839ebd78-f465-4354-8c38-e036459f2114 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:31.573088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839ebd78-f465-4354-8c38-e036459f2114 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:31.573559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839ebd78-f465-4354-8c38-e036459f2114 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:31.592580 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-839ebd78-f465-4354-8c38-e036459f2114 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:31.594144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-839ebd78-f465-4354-8c38-e036459f2114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:31.625112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:31.666374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 413/834] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:31 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:31.937599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378b875f-2a4e-4334-be54-06f783f9e6b9 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:32.001630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cba173d-2c35-4bf7-9e97-ba3691be65b6 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:32.001921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cba173d-2c35-4bf7-9e97-ba3691be65b6 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:32.002242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cba173d-2c35-4bf7-9e97-ba3691be65b6 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:32.003831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 414/835] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:32.008911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0242dc0-7d00-4bf9-a6a6-188134fedab6 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:32.009191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0242dc0-7d00-4bf9-a6a6-188134fedab6 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:32.009462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0242dc0-7d00-4bf9-a6a6-188134fedab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:32.030459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-f0242dc0-7d00-4bf9-a6a6-188134fedab6 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:32.031954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f0242dc0-7d00-4bf9-a6a6-188134fedab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:32.046366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378b875f-2a4e-4334-be54-06f783f9e6b9 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:32.083684 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba589c-e30a-4125-9042-9fa95c9869f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:32.084416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba589c-e30a-4125-9042-9fa95c9869f4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:32.084688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba589c-e30a-4125-9042-9fa95c9869f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:32.084955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba589c-e30a-4125-9042-9fa95c9869f4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:32.099614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ba589c-e30a-4125-9042-9fa95c9869f4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-a4ba589c-e30a-4125-9042-9fa95c9869f4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:32.106514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:32.107774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:17:32.108813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:32.111563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 422/836] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:17:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:32.119671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba589c-e30a-4125-9042-9fa95c9869f4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:32.124242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 415/837] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:32 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:32.147473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:32.228110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba589c-e30a-4125-9042-9fa95c9869f4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:32.262175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a346e3-b218-43a7-8b52-36e83709c3e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:32.263061 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a346e3-b218-43a7-8b52-36e83709c3e7 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:32.263530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a346e3-b218-43a7-8b52-36e83709c3e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:32.263986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a346e3-b218-43a7-8b52-36e83709c3e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:32.264832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a346e3-b218-43a7-8b52-36e83709c3e7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:32.265379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a346e3-b218-43a7-8b52-36e83709c3e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:32.266999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a346e3-b218-43a7-8b52-36e83709c3e7 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:32.275448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 423/838] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:17:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:32.283476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0242dc0-7d00-4bf9-a6a6-188134fedab6 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:32.382543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:32.392212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0242dc0-7d00-4bf9-a6a6-188134fedab6 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:32.407267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a346e3-b218-43a7-8b52-36e83709c3e7 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:32.435594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901fc8b2-4f74-4201-8eee-329c28ca5b5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:32.435594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901fc8b2-4f74-4201-8eee-329c28ca5b5e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:32.435594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901fc8b2-4f74-4201-8eee-329c28ca5b5e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:32.435594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901fc8b2-4f74-4201-8eee-329c28ca5b5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:32.441610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901fc8b2-4f74-4201-8eee-329c28ca5b5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:32.443527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901fc8b2-4f74-4201-8eee-329c28ca5b5e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-901fc8b2-4f74-4201-8eee-329c28ca5b5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:32.458899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:32.460091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:32.461007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:32.461007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:32.461007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:32.461765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 416/839] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:32.467069 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901fc8b2-4f74-4201-8eee-329c28ca5b5e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:32.513156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:32.528926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a346e3-b218-43a7-8b52-36e83709c3e7 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:32.567381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0254a98-bb56-4e54-9170-911fc12d8d43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:32.568033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0254a98-bb56-4e54-9170-911fc12d8d43 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:32.568310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0254a98-bb56-4e54-9170-911fc12d8d43 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:32.568576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0254a98-bb56-4e54-9170-911fc12d8d43 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:32.569168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0254a98-bb56-4e54-9170-911fc12d8d43 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:32.570790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0254a98-bb56-4e54-9170-911fc12d8d43 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-d0254a98-bb56-4e54-9170-911fc12d8d43 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:32.580486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:32.581617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:17:32.582924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:32.586371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901fc8b2-4f74-4201-8eee-329c28ca5b5e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:32.586933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 424/840] 10.222.0.16 () {56 vars in 1132 bytes} [Tue Sep 13 15:17:32 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:32.587850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0254a98-bb56-4e54-9170-911fc12d8d43 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:32.621912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474ebd3e-1518-48d2-9749-30bc677acfdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:32.623028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ebd3e-1518-48d2-9749-30bc677acfdd None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:32.623490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ebd3e-1518-48d2-9749-30bc677acfdd None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:32.623938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ebd3e-1518-48d2-9749-30bc677acfdd None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:32.624774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474ebd3e-1518-48d2-9749-30bc677acfdd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:32.625253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474ebd3e-1518-48d2-9749-30bc677acfdd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:32.626846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474ebd3e-1518-48d2-9749-30bc677acfdd None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:32.650144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 417/841] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:32.650144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474ebd3e-1518-48d2-9749-30bc677acfdd None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:32.658705 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:32.704492 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0254a98-bb56-4e54-9170-911fc12d8d43 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:32.742760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0b317c-7c64-48df-b028-1da40503f5d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:32.743653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0b317c-7c64-48df-b028-1da40503f5d9 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:32.744070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0b317c-7c64-48df-b028-1da40503f5d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:32.744488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0b317c-7c64-48df-b028-1da40503f5d9 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:32.745383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0b317c-7c64-48df-b028-1da40503f5d9 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:32.745907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0b317c-7c64-48df-b028-1da40503f5d9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:32.747467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0b317c-7c64-48df-b028-1da40503f5d9 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:32.751447 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474ebd3e-1518-48d2-9749-30bc677acfdd None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:32.757435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 425/842] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:32 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:32.758546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0b317c-7c64-48df-b028-1da40503f5d9 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:32.781340 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:32.789512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13ab863-b675-4b7b-bfdc-5b9aaae1800c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:32.790369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ab863-b675-4b7b-bfdc-5b9aaae1800c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:32.790646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ab863-b675-4b7b-bfdc-5b9aaae1800c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:32.790911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ab863-b675-4b7b-bfdc-5b9aaae1800c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:32.800861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13ab863-b675-4b7b-bfdc-5b9aaae1800c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:32.803535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13ab863-b675-4b7b-bfdc-5b9aaae1800c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-e13ab863-b675-4b7b-bfdc-5b9aaae1800c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:32.819526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:32.820716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:32.821798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 418/843] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Sep 13 15:17:32 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:32.830066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13ab863-b675-4b7b-bfdc-5b9aaae1800c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:32.878432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0b317c-7c64-48df-b028-1da40503f5d9 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:32.912408 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:32.914005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ae813e-5a05-4bc2-91a3-dc9799d72871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:32.914811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae813e-5a05-4bc2-91a3-dc9799d72871 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:32.915254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae813e-5a05-4bc2-91a3-dc9799d72871 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:32.915676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae813e-5a05-4bc2-91a3-dc9799d72871 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:32.916486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ae813e-5a05-4bc2-91a3-dc9799d72871 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:32.916764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ae813e-5a05-4bc2-91a3-dc9799d72871 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:32.918161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ae813e-5a05-4bc2-91a3-dc9799d72871 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:32.929820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 426/844] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:32 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:32.953566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13ab863-b675-4b7b-bfdc-5b9aaae1800c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:32.998944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d781735-1830-4c21-9205-60b300e6240c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:32.999915 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d781735-1830-4c21-9205-60b300e6240c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.000379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d781735-1830-4c21-9205-60b300e6240c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.000879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d781735-1830-4c21-9205-60b300e6240c None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.001801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d781735-1830-4c21-9205-60b300e6240c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:33.002316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d781735-1830-4c21-9205-60b300e6240c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:33.004207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d781735-1830-4c21-9205-60b300e6240c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:33.026341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 419/845] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Sep 13 15:17:32 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:33.035431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ae813e-5a05-4bc2-91a3-dc9799d72871 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:33.122722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ee99c1-c372-4110-b003-23f216eca66e None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.123023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:33.123554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ee99c1-c372-4110-b003-23f216eca66e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.124017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ee99c1-c372-4110-b003-23f216eca66e None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.125606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 420/846] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:33.131224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a53de-c582-4f78-9d8d-df8aa9f9987f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.131792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a53de-c582-4f78-9d8d-df8aa9f9987f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.132064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a53de-c582-4f78-9d8d-df8aa9f9987f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.146335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ae813e-5a05-4bc2-91a3-dc9799d72871 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:33.155756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c00a53de-c582-4f78-9d8d-df8aa9f9987f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:33.156945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c00a53de-c582-4f78-9d8d-df8aa9f9987f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:33.182311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fe0673-eb5a-4dd5-b10f-c7aec6b96554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:33.182930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fe0673-eb5a-4dd5-b10f-c7aec6b96554 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.183375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fe0673-eb5a-4dd5-b10f-c7aec6b96554 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.183653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fe0673-eb5a-4dd5-b10f-c7aec6b96554 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.230941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 421/847] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:33 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:33.261241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fe0673-eb5a-4dd5-b10f-c7aec6b96554 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:33.262791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fe0673-eb5a-4dd5-b10f-c7aec6b96554 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:33.351276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:33.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 427/848] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:33 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:17:33.379531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00a53de-c582-4f78-9d8d-df8aa9f9987f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:33.415658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027689fc-383f-486a-af7e-36001726f43d None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.415987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027689fc-383f-486a-af7e-36001726f43d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.416237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027689fc-383f-486a-af7e-36001726f43d None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.419195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 428/849] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:33.423038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a80c05-8a5a-45c0-974e-709343abec6f None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.423318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a80c05-8a5a-45c0-974e-709343abec6f None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.423589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a80c05-8a5a-45c0-974e-709343abec6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.438761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:33.443727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-76a80c05-8a5a-45c0-974e-709343abec6f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:33.445010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-76a80c05-8a5a-45c0-974e-709343abec6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:33.486847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00a53de-c582-4f78-9d8d-df8aa9f9987f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:33.520604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 429/850] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:33 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:33.521753 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a80c05-8a5a-45c0-974e-709343abec6f None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:33.527956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b422c34f-ee47-40a3-a7f1-f00066b066be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:33.528921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b422c34f-ee47-40a3-a7f1-f00066b066be None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.529494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b422c34f-ee47-40a3-a7f1-f00066b066be None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.529941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b422c34f-ee47-40a3-a7f1-f00066b066be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.544647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b422c34f-ee47-40a3-a7f1-f00066b066be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:33.547382 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b422c34f-ee47-40a3-a7f1-f00066b066be None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-b422c34f-ee47-40a3-a7f1-f00066b066be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:33.571651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:33.572676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:33.573600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:33.573600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:33.573600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:33.580596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 422/851] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:33.581831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b422c34f-ee47-40a3-a7f1-f00066b066be None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:33.626610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a80c05-8a5a-45c0-974e-709343abec6f None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:33.640102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:33.668958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dad2ef-ab54-42a0-a449-bd06c0cb45b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:33.669839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dad2ef-ab54-42a0-a449-bd06c0cb45b1 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.670284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dad2ef-ab54-42a0-a449-bd06c0cb45b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.670652 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dad2ef-ab54-42a0-a449-bd06c0cb45b1 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.671330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dad2ef-ab54-42a0-a449-bd06c0cb45b1 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:33.672802 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dad2ef-ab54-42a0-a449-bd06c0cb45b1 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-08dad2ef-ab54-42a0-a449-bd06c0cb45b1 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:33.682394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:33.683526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:33.684356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 430/852] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:17:33 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:33.690837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dad2ef-ab54-42a0-a449-bd06c0cb45b1 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:33.709443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b422c34f-ee47-40a3-a7f1-f00066b066be None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:33.720431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:33.745828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa300583-4614-4673-ab61-d859918730bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:33.746492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa300583-4614-4673-ab61-d859918730bb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.746764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa300583-4614-4673-ab61-d859918730bb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.747244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa300583-4614-4673-ab61-d859918730bb None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.747930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa300583-4614-4673-ab61-d859918730bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:33.748062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa300583-4614-4673-ab61-d859918730bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:33.749402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa300583-4614-4673-ab61-d859918730bb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:33.766002 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 423/853] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:33.767094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa300583-4614-4673-ab61-d859918730bb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:33.826594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dad2ef-ab54-42a0-a449-bd06c0cb45b1 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:33.863098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d81431-8950-4301-ac5c-80106f5d6a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:33.864070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d81431-8950-4301-ac5c-80106f5d6a34 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.864347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d81431-8950-4301-ac5c-80106f5d6a34 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.864610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d81431-8950-4301-ac5c-80106f5d6a34 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.865249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d81431-8950-4301-ac5c-80106f5d6a34 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:33.865548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d81431-8950-4301-ac5c-80106f5d6a34 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:33.867003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d81431-8950-4301-ac5c-80106f5d6a34 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:33.872442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 431/854] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:17:33 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:33.873077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa300583-4614-4673-ab61-d859918730bb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:33.873649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d81431-8950-4301-ac5c-80106f5d6a34 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:33.911179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30033a0f-e2b8-4183-821a-baca6b963e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:33.911820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30033a0f-e2b8-4183-821a-baca6b963e27 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:33.912097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30033a0f-e2b8-4183-821a-baca6b963e27 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:33.912366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30033a0f-e2b8-4183-821a-baca6b963e27 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:33.917918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:33.919150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30033a0f-e2b8-4183-821a-baca6b963e27 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:33.920617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30033a0f-e2b8-4183-821a-baca6b963e27 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-30033a0f-e2b8-4183-821a-baca6b963e27 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:33.931900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:33.932968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:33.934181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:33.934181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:33.934181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:33.934181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:33.934181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:33.934181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:33.934181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:33.934181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:33.937910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 424/855] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:33.938809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30033a0f-e2b8-4183-821a-baca6b963e27 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:33.967788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:33.985906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d81431-8950-4301-ac5c-80106f5d6a34 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:34.023899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da0efea-47e6-4e93-947e-fa536d0fc460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:34.024531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da0efea-47e6-4e93-947e-fa536d0fc460 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:34.024803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da0efea-47e6-4e93-947e-fa536d0fc460 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:34.025070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da0efea-47e6-4e93-947e-fa536d0fc460 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:34.035142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da0efea-47e6-4e93-947e-fa536d0fc460 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-7da0efea-47e6-4e93-947e-fa536d0fc460 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:34.039903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:34.042531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 432/856] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:34.043408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da0efea-47e6-4e93-947e-fa536d0fc460 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:34.050955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30033a0f-e2b8-4183-821a-baca6b963e27 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:34.087110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42b50e-b0ec-429c-bd4c-612f7f3541f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:34.087741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42b50e-b0ec-429c-bd4c-612f7f3541f5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:34.088306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42b50e-b0ec-429c-bd4c-612f7f3541f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:34.088589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42b50e-b0ec-429c-bd4c-612f7f3541f5 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:34.089231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c42b50e-b0ec-429c-bd4c-612f7f3541f5 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:34.089535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c42b50e-b0ec-429c-bd4c-612f7f3541f5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:34.091025 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c42b50e-b0ec-429c-bd4c-612f7f3541f5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:34.095995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 425/857] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:17:33 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:34.096802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42b50e-b0ec-429c-bd4c-612f7f3541f5 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:34.149770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:34.154507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da0efea-47e6-4e93-947e-fa536d0fc460 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:34.200844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe99e2-3297-4b58-bf7f-a496002c452b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:34.200844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe99e2-3297-4b58-bf7f-a496002c452b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:34.200844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe99e2-3297-4b58-bf7f-a496002c452b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:34.200844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe99e2-3297-4b58-bf7f-a496002c452b None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:34.201288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fe99e2-3297-4b58-bf7f-a496002c452b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:34.201654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fe99e2-3297-4b58-bf7f-a496002c452b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:34.203137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fe99e2-3297-4b58-bf7f-a496002c452b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:34.221387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 433/858] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:34.222987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe99e2-3297-4b58-bf7f-a496002c452b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:34.240563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:34.250769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42b50e-b0ec-429c-bd4c-612f7f3541f5 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:34.287918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ecfec3-f5f8-4bf7-a824-9d0f03f4afd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:34.288546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ecfec3-f5f8-4bf7-a824-9d0f03f4afd8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:34.288820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ecfec3-f5f8-4bf7-a824-9d0f03f4afd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:34.289088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ecfec3-f5f8-4bf7-a824-9d0f03f4afd8 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:34.289759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ecfec3-f5f8-4bf7-a824-9d0f03f4afd8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:34.290054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ecfec3-f5f8-4bf7-a824-9d0f03f4afd8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:34.291685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ecfec3-f5f8-4bf7-a824-9d0f03f4afd8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:34.306662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 426/859] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:34.308114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ecfec3-f5f8-4bf7-a824-9d0f03f4afd8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:34.349200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe99e2-3297-4b58-bf7f-a496002c452b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:34.383873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccad909-4c56-4b93-a7ea-5749568ea457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:34.384484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccad909-4c56-4b93-a7ea-5749568ea457 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:34.384756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccad909-4c56-4b93-a7ea-5749568ea457 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:34.385026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccad909-4c56-4b93-a7ea-5749568ea457 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:34.385743 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccad909-4c56-4b93-a7ea-5749568ea457 None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=eac26eaa3b1a4c8c849a45ed9ba20301) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:34.386048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccad909-4c56-4b93-a7ea-5749568ea457 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:34.387495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccad909-4c56-4b93-a7ea-5749568ea457 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:34.394081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 434/860] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:34 2022] GET /identity/v3/role_assignments?user.id=eac26eaa3b1a4c8c849a45ed9ba20301&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:34.440857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ecfec3-f5f8-4bf7-a824-9d0f03f4afd8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:34.477064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6feb51c-a860-4b1b-8c44-fd5f6370f03a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:34.477635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6feb51c-a860-4b1b-8c44-fd5f6370f03a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:34.477953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6feb51c-a860-4b1b-8c44-fd5f6370f03a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:34.478257 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6feb51c-a860-4b1b-8c44-fd5f6370f03a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:34.490618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6feb51c-a860-4b1b-8c44-fd5f6370f03a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:34.492173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6feb51c-a860-4b1b-8c44-fd5f6370f03a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-f6feb51c-a860-4b1b-8c44-fd5f6370f03a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:34.508425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:34.509432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:34.510318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:34.510318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:34.510318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:34.514243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 427/861] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:34.516068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccad909-4c56-4b93-a7ea-5749568ea457 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:34.544827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:34.624835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccad909-4c56-4b93-a7ea-5749568ea457 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:34.660767 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ab6b9f-74a9-49e1-ad90-e31cca4c8471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:34.661383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ab6b9f-74a9-49e1-ad90-e31cca4c8471 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:34.661656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ab6b9f-74a9-49e1-ad90-e31cca4c8471 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:34.661923 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ab6b9f-74a9-49e1-ad90-e31cca4c8471 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:34.662596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ab6b9f-74a9-49e1-ad90-e31cca4c8471 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:34.662896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ab6b9f-74a9-49e1-ad90-e31cca4c8471 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:34.664272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ab6b9f-74a9-49e1-ad90-e31cca4c8471 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:34.676911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 435/862] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:34.682827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6feb51c-a860-4b1b-8c44-fd5f6370f03a None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:34.693695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:34.787473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6feb51c-a860-4b1b-8c44-fd5f6370f03a None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:34.822598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bc525b-b857-456b-8b72-a76f36e0b9ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:34.823245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bc525b-b857-456b-8b72-a76f36e0b9ed None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:34.823520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bc525b-b857-456b-8b72-a76f36e0b9ed None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:34.823796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bc525b-b857-456b-8b72-a76f36e0b9ed None admin] PATH_INFO: `/v3/users/swift` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:34.832055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bc525b-b857-456b-8b72-a76f36e0b9ed None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:34.833528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bc525b-b857-456b-8b72-a76f36e0b9ed None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-d5bc525b-b857-456b-8b72-a76f36e0b9ed None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:34.845159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:34.846088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:34.847052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 428/863] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:17:34 2022] GET /identity/v3/users/swift => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:34.855234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ab6b9f-74a9-49e1-ad90-e31cca4c8471 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:34.939102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:34.959833 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ab6b9f-74a9-49e1-ad90-e31cca4c8471 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:34.993993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5726b0ec-1ef7-487a-b46d-a8ed9f218ce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:34.994649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726b0ec-1ef7-487a-b46d-a8ed9f218ce4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:34.994926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726b0ec-1ef7-487a-b46d-a8ed9f218ce4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:34.995196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726b0ec-1ef7-487a-b46d-a8ed9f218ce4 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:34.995924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5726b0ec-1ef7-487a-b46d-a8ed9f218ce4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:34.996231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5726b0ec-1ef7-487a-b46d-a8ed9f218ce4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:34.997922 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5726b0ec-1ef7-487a-b46d-a8ed9f218ce4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:35.015729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 436/864] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Sep 13 15:17:34 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:35.021517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bc525b-b857-456b-8b72-a76f36e0b9ed None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:35.105659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:35.143290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bc525b-b857-456b-8b72-a76f36e0b9ed None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:35.162137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a9efc6-c5ab-443e-b79a-5ea371785fd5 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:35.162654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a9efc6-c5ab-443e-b79a-5ea371785fd5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:35.162654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a9efc6-c5ab-443e-b79a-5ea371785fd5 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:35.164113 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 437/865] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:35.169318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ddda5-ab8c-42a0-9ac2-a7642177dcdf None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:35.169730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ddda5-ab8c-42a0-9ac2-a7642177dcdf None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:35.170002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ddda5-ab8c-42a0-9ac2-a7642177dcdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:35.185812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5304cec-4d51-4099-865c-838edf322f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:35.186791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5304cec-4d51-4099-865c-838edf322f01 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:35.187347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5304cec-4d51-4099-865c-838edf322f01 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:35.187810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5304cec-4d51-4099-865c-838edf322f01 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:35.189038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7f3ddda5-ab8c-42a0-9ac2-a7642177dcdf None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:35.190314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7f3ddda5-ab8c-42a0-9ac2-a7642177dcdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:35.266173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 438/866] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:35 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:35.281802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5304cec-4d51-4099-865c-838edf322f01 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:35.283532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5304cec-4d51-4099-865c-838edf322f01 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:35.349446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:35.390024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 429/867] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:35 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:17:35.395690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3ddda5-ab8c-42a0-9ac2-a7642177dcdf None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:35.469030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:35.503005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3ddda5-ab8c-42a0-9ac2-a7642177dcdf None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:35.536597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aae39a1-f06f-4075-a885-a56bd844b56c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:35.537228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aae39a1-f06f-4075-a885-a56bd844b56c None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:35.537499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aae39a1-f06f-4075-a885-a56bd844b56c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:35.537763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aae39a1-f06f-4075-a885-a56bd844b56c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:35.548327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aae39a1-f06f-4075-a885-a56bd844b56c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:35.550676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aae39a1-f06f-4075-a885-a56bd844b56c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-6aae39a1-f06f-4075-a885-a56bd844b56c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:35.567233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:35.568220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:35.569212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:35.569212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:35.569212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:35.569212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 439/868] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:35.574200 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5304cec-4d51-4099-865c-838edf322f01 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:35.581894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aae39a1-f06f-4075-a885-a56bd844b56c None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:35.701751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5304cec-4d51-4099-865c-838edf322f01 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:35.707305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aae39a1-f06f-4075-a885-a56bd844b56c None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:35.734678 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:35.742836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffde27e1-ecd7-40a8-8eab-20b281276d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:35.743609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffde27e1-ecd7-40a8-8eab-20b281276d25 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:35.743887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffde27e1-ecd7-40a8-8eab-20b281276d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:35.744154 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffde27e1-ecd7-40a8-8eab-20b281276d25 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:35.745085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffde27e1-ecd7-40a8-8eab-20b281276d25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:35.745395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffde27e1-ecd7-40a8-8eab-20b281276d25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:35.747364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffde27e1-ecd7-40a8-8eab-20b281276d25 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:35.751289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f79a5b-e810-4278-8667-608f2794008f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:35.752223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f79a5b-e810-4278-8667-608f2794008f None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:35.752756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f79a5b-e810-4278-8667-608f2794008f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:35.753222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f79a5b-e810-4278-8667-608f2794008f None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:35.754073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f79a5b-e810-4278-8667-608f2794008f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:35.756264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f79a5b-e810-4278-8667-608f2794008f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:35.762180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 430/869] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:35.766451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffde27e1-ecd7-40a8-8eab-20b281276d25 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:35.786157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 440/870] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:17:35 2022] POST /identity/v3/services => generated 269 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:17:35.810824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:35.898128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffde27e1-ecd7-40a8-8eab-20b281276d25 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:35.942335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a9b80f-ad76-4b62-82c1-3717591a8b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:35.942987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a9b80f-ad76-4b62-82c1-3717591a8b45 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:35.943272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a9b80f-ad76-4b62-82c1-3717591a8b45 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:35.943586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a9b80f-ad76-4b62-82c1-3717591a8b45 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:35.951651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a9b80f-ad76-4b62-82c1-3717591a8b45 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:35.953139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a9b80f-ad76-4b62-82c1-3717591a8b45 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-d5a9b80f-ad76-4b62-82c1-3717591a8b45 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:35.965728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:35.967368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:35.968277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:35.968277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:35.968277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:35.968277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:35.968277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:35.968277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:35.968277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:35.968277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:35.971928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 431/871] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:35.973339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f79a5b-e810-4278-8667-608f2794008f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:36.080330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f79a5b-e810-4278-8667-608f2794008f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:36.114921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206ccc94-cf3e-47c5-a8d1-ed6998f1e4a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:36.115538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206ccc94-cf3e-47c5-a8d1-ed6998f1e4a6 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:36.115811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206ccc94-cf3e-47c5-a8d1-ed6998f1e4a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:36.116076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206ccc94-cf3e-47c5-a8d1-ed6998f1e4a6 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:36.116765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206ccc94-cf3e-47c5-a8d1-ed6998f1e4a6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:36.117064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206ccc94-cf3e-47c5-a8d1-ed6998f1e4a6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:36.118572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206ccc94-cf3e-47c5-a8d1-ed6998f1e4a6 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:36.136099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 441/872] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:36.137691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a9b80f-ad76-4b62-82c1-3717591a8b45 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:36.221275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:36.242148 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a9b80f-ad76-4b62-82c1-3717591a8b45 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:36.279865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54275232-ec10-48ce-87c2-b6ad6d0c94cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:36.280502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54275232-ec10-48ce-87c2-b6ad6d0c94cc None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:36.280850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54275232-ec10-48ce-87c2-b6ad6d0c94cc None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:36.281117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54275232-ec10-48ce-87c2-b6ad6d0c94cc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:36.281864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54275232-ec10-48ce-87c2-b6ad6d0c94cc None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=35ece665f492449282ab03ad91f0b254) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:36.282202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54275232-ec10-48ce-87c2-b6ad6d0c94cc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:36.283694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54275232-ec10-48ce-87c2-b6ad6d0c94cc None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:36.291332 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 432/873] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:36 2022] GET /identity/v3/role_assignments?user.id=35ece665f492449282ab03ad91f0b254&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=22819414415942d399db534708fe7103 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:36.404056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8d2f97-4c25-4249-ab22-9b4db8852895 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:36.404627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8d2f97-4c25-4249-ab22-9b4db8852895 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:36.404722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8d2f97-4c25-4249-ab22-9b4db8852895 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:36.406169 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 442/874] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:36.411881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142b7ce8-1173-477a-9b11-2b51eae6e65c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:36.412550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142b7ce8-1173-477a-9b11-2b51eae6e65c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:36.413014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142b7ce8-1173-477a-9b11-2b51eae6e65c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:36.433275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-142b7ce8-1173-477a-9b11-2b51eae6e65c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:36.434959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-142b7ce8-1173-477a-9b11-2b51eae6e65c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:36.509951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 433/875] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:36 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:36.770731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a517e012-cdb9-4e87-9a6c-d2a5b5f93ac9 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:36.770912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a517e012-cdb9-4e87-9a6c-d2a5b5f93ac9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:36.771187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a517e012-cdb9-4e87-9a6c-d2a5b5f93ac9 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:36.772849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 443/876] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:36.778177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ad2435-670e-44c5-823f-9cd54711243a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:36.778477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ad2435-670e-44c5-823f-9cd54711243a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:36.778846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ad2435-670e-44c5-823f-9cd54711243a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:36.794160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a517e012-cdb9-4e87-9a6c-d2a5b5f93ac9 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:36.799479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-81ad2435-670e-44c5-823f-9cd54711243a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:36.800698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-81ad2435-670e-44c5-823f-9cd54711243a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:36.847625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:36.895755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 434/877] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:36 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:36.926808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a517e012-cdb9-4e87-9a6c-d2a5b5f93ac9 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:36.961960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be37a61a-d985-40a7-8c3a-8d6fef8417c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:36.962604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be37a61a-d985-40a7-8c3a-8d6fef8417c5 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:36.962878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be37a61a-d985-40a7-8c3a-8d6fef8417c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:36.963142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be37a61a-d985-40a7-8c3a-8d6fef8417c5 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:36.963790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be37a61a-d985-40a7-8c3a-8d6fef8417c5 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:36.965148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be37a61a-d985-40a7-8c3a-8d6fef8417c5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:36.983908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 444/878] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:17:36 2022] POST /identity/v3/services => generated 251 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:17:37.175356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ad2435-670e-44c5-823f-9cd54711243a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:37.184772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:37.281482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ad2435-670e-44c5-823f-9cd54711243a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:37.323917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdbf95b-2d69-4a07-af9e-a0b4ca51f066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:37.324556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdbf95b-2d69-4a07-af9e-a0b4ca51f066 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:37.324827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdbf95b-2d69-4a07-af9e-a0b4ca51f066 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:37.325125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdbf95b-2d69-4a07-af9e-a0b4ca51f066 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:37.335286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdbf95b-2d69-4a07-af9e-a0b4ca51f066 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-8bdbf95b-2d69-4a07-af9e-a0b4ca51f066 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:37.339966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:37.341018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:37.341862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 435/879] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Sep 13 15:17:37 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:37.347129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be37a61a-d985-40a7-8c3a-8d6fef8417c5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:37.455776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be37a61a-d985-40a7-8c3a-8d6fef8417c5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:37.457170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:37.492794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6934d5-4e51-4f73-9185-bf5ef134a527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:37.493410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6934d5-4e51-4f73-9185-bf5ef134a527 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:37.493682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6934d5-4e51-4f73-9185-bf5ef134a527 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:37.493953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6934d5-4e51-4f73-9185-bf5ef134a527 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:37.494659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6934d5-4e51-4f73-9185-bf5ef134a527 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:37.494959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6934d5-4e51-4f73-9185-bf5ef134a527 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:37.496327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6934d5-4e51-4f73-9185-bf5ef134a527 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:37.505301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 445/880] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Sep 13 15:17:37 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:37.511264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdbf95b-2d69-4a07-af9e-a0b4ca51f066 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:37.579829 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:37.620737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdbf95b-2d69-4a07-af9e-a0b4ca51f066 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:37.656436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c341b5-9ea7-4c83-8971-5b842a5648ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:37.657082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c341b5-9ea7-4c83-8971-5b842a5648ba None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:37.657357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c341b5-9ea7-4c83-8971-5b842a5648ba None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:37.657624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c341b5-9ea7-4c83-8971-5b842a5648ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:37.667976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c341b5-9ea7-4c83-8971-5b842a5648ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:37.669870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c341b5-9ea7-4c83-8971-5b842a5648ba None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-97c341b5-9ea7-4c83-8971-5b842a5648ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:37.684919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:37.685873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:37.686780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:37.686780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:37.686780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:37.686780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 436/881] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:37.691893 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6934d5-4e51-4f73-9185-bf5ef134a527 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:37.799441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6934d5-4e51-4f73-9185-bf5ef134a527 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:37.833791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8549f810-2d5b-4165-ab88-6243ed1f5ed5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:37.834434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8549f810-2d5b-4165-ab88-6243ed1f5ed5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:37.834709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8549f810-2d5b-4165-ab88-6243ed1f5ed5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:37.834979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8549f810-2d5b-4165-ab88-6243ed1f5ed5 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:37.835621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8549f810-2d5b-4165-ab88-6243ed1f5ed5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:37.835926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8549f810-2d5b-4165-ab88-6243ed1f5ed5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:37.837300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8549f810-2d5b-4165-ab88-6243ed1f5ed5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:37.851827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 446/882] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:37.857633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c341b5-9ea7-4c83-8971-5b842a5648ba None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:37.959675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c341b5-9ea7-4c83-8971-5b842a5648ba None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:37.975703 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:38.001455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf780c8-38e6-40b8-9008-3d9b45034642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:38.002144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf780c8-38e6-40b8-9008-3d9b45034642 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.002429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf780c8-38e6-40b8-9008-3d9b45034642 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.002698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf780c8-38e6-40b8-9008-3d9b45034642 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.011708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf780c8-38e6-40b8-9008-3d9b45034642 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:38.013248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf780c8-38e6-40b8-9008-3d9b45034642 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-6cf780c8-38e6-40b8-9008-3d9b45034642 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:38.024898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:38.025927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:38.026972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 437/883] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Sep 13 15:17:37 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:38.032207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8549f810-2d5b-4165-ab88-6243ed1f5ed5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:38.057908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:38.095733 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f5102f-ffd5-462c-afe2-3f3b16f97afb None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.096103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f5102f-ffd5-462c-afe2-3f3b16f97afb None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.096295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f5102f-ffd5-462c-afe2-3f3b16f97afb None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.097864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 438/884] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:38.102939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49549159-1ac4-4c6a-9117-4db4970d0f2f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.103229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49549159-1ac4-4c6a-9117-4db4970d0f2f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.103500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49549159-1ac4-4c6a-9117-4db4970d0f2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.123121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-49549159-1ac4-4c6a-9117-4db4970d0f2f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:38.124326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-49549159-1ac4-4c6a-9117-4db4970d0f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:38.138098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8549f810-2d5b-4165-ab88-6243ed1f5ed5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:38.174279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a87295-aab9-4d74-8020-c7f3d84aeca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:38.174906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a87295-aab9-4d74-8020-c7f3d84aeca5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.175186 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a87295-aab9-4d74-8020-c7f3d84aeca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.175457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a87295-aab9-4d74-8020-c7f3d84aeca5 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.176196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a87295-aab9-4d74-8020-c7f3d84aeca5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:38.176502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a87295-aab9-4d74-8020-c7f3d84aeca5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:38.177938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a87295-aab9-4d74-8020-c7f3d84aeca5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:38.201627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 447/885] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Sep 13 15:17:38 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:38.202826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a87295-aab9-4d74-8020-c7f3d84aeca5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:38.219704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 439/886] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:38 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:38.296571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:38.315760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a87295-aab9-4d74-8020-c7f3d84aeca5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:38.350623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f897c-42be-41d3-acec-87557664953f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:38.351331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f897c-42be-41d3-acec-87557664953f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.351610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f897c-42be-41d3-acec-87557664953f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.351883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f897c-42be-41d3-acec-87557664953f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.420534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2f897c-42be-41d3-acec-87557664953f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:38.421988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2f897c-42be-41d3-acec-87557664953f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:38.494150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49549159-1ac4-4c6a-9117-4db4970d0f2f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:38.516866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:38.544423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 448/887] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:38 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 13 15:17:38.545470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f897c-42be-41d3-acec-87557664953f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:38.619752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49549159-1ac4-4c6a-9117-4db4970d0f2f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:38.622841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:38.657231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f897c-42be-41d3-acec-87557664953f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:38.660531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b17b56-5198-4a4a-abeb-481bc9c34b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:38.661189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b17b56-5198-4a4a-abeb-481bc9c34b94 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.661468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b17b56-5198-4a4a-abeb-481bc9c34b94 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.661910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b17b56-5198-4a4a-abeb-481bc9c34b94 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.662535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b17b56-5198-4a4a-abeb-481bc9c34b94 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:38.663918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b17b56-5198-4a4a-abeb-481bc9c34b94 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-06b17b56-5198-4a4a-abeb-481bc9c34b94 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:38.673381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:38.674393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:38.675295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 440/888] 10.222.0.16 () {56 vars in 1132 bytes} [Tue Sep 13 15:17:38 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:38.676114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ee732b-0467-49a6-9b8f-0982f84c7091 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.676871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ee732b-0467-49a6-9b8f-0982f84c7091 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.676871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ee732b-0467-49a6-9b8f-0982f84c7091 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.677898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 441/889] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:38 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:38.682869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ee732b-0467-49a6-9b8f-0982f84c7091 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:38.697464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f85660a-d6d7-4858-bf2b-05c51e09a94a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:38.698154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f85660a-d6d7-4858-bf2b-05c51e09a94a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.698399 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f85660a-d6d7-4858-bf2b-05c51e09a94a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.698695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f85660a-d6d7-4858-bf2b-05c51e09a94a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.709200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f85660a-d6d7-4858-bf2b-05c51e09a94a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:38.710666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f85660a-d6d7-4858-bf2b-05c51e09a94a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-9f85660a-d6d7-4858-bf2b-05c51e09a94a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:38.726374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:38.727349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:38.728265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:38.728265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:38.728265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:38.728265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 449/890] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:38.728932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a0006-d061-4b0d-86db-af2f935a244d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.728932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a0006-d061-4b0d-86db-af2f935a244d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.729492 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a0006-d061-4b0d-86db-af2f935a244d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.748245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-397a0006-d061-4b0d-86db-af2f935a244d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:38.749453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-397a0006-d061-4b0d-86db-af2f935a244d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:38.778779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:38.794365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ee732b-0467-49a6-9b8f-0982f84c7091 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:38.820820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 450/891] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:38 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:38.827070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397a0006-d061-4b0d-86db-af2f935a244d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:38.831262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d9f3c-8532-4896-964b-66b189d2a331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:38.832153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d9f3c-8532-4896-964b-66b189d2a331 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.832648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d9f3c-8532-4896-964b-66b189d2a331 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.833127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d9f3c-8532-4896-964b-66b189d2a331 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.833959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d9f3c-8532-4896-964b-66b189d2a331 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:38.834482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d9f3c-8532-4896-964b-66b189d2a331 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:38.836084 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07d9f3c-8532-4896-964b-66b189d2a331 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:38.841961 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 442/892] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:38 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:38.847481 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d9f3c-8532-4896-964b-66b189d2a331 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:38.882928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:38.954559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d9f3c-8532-4896-964b-66b189d2a331 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:38.969005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397a0006-d061-4b0d-86db-af2f935a244d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:38.989363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e48de74-1ffb-4b66-93d2-268299b3581e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:38.990062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48de74-1ffb-4b66-93d2-268299b3581e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:38.990313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48de74-1ffb-4b66-93d2-268299b3581e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:38.990594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e48de74-1ffb-4b66-93d2-268299b3581e None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:38.991256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e48de74-1ffb-4b66-93d2-268299b3581e None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:38.991563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e48de74-1ffb-4b66-93d2-268299b3581e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:38.993002 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e48de74-1ffb-4b66-93d2-268299b3581e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:39.004707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 443/893] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:38 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:39.007268 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e48de74-1ffb-4b66-93d2-268299b3581e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:39.016805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e954fd-1487-4920-898f-f58aac03f08a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:39.017470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e954fd-1487-4920-898f-f58aac03f08a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.017750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e954fd-1487-4920-898f-f58aac03f08a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.018015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e954fd-1487-4920-898f-f58aac03f08a None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.018687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e954fd-1487-4920-898f-f58aac03f08a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:39.019031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e954fd-1487-4920-898f-f58aac03f08a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:39.020539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e954fd-1487-4920-898f-f58aac03f08a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:39.033049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 451/894] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:39.038754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e954fd-1487-4920-898f-f58aac03f08a None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:39.069121 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:39.123177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:39.129812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e48de74-1ffb-4b66-93d2-268299b3581e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:39.150619 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e954fd-1487-4920-898f-f58aac03f08a None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:39.164453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3054d72-cc50-4791-8017-b06c02e28f10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:39.165320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3054d72-cc50-4791-8017-b06c02e28f10 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.165320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3054d72-cc50-4791-8017-b06c02e28f10 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.165579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3054d72-cc50-4791-8017-b06c02e28f10 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.166303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3054d72-cc50-4791-8017-b06c02e28f10 None admin] RBAC: Adding query filter params (interface=public, service_id=5bf71884275a4f189c3b5e2cc7bec88d, region_id=RegionOne) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:39.166637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3054d72-cc50-4791-8017-b06c02e28f10 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:39.168030 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3054d72-cc50-4791-8017-b06c02e28f10 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:39.177532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 444/895] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Sep 13 15:17:39 2022] GET /identity/v3/endpoints?service_id=5bf71884275a4f189c3b5e2cc7bec88d&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:39.177532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3054d72-cc50-4791-8017-b06c02e28f10 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:39.192961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871df9ce-3cc5-4fe5-b405-e6532f47a49d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:39.193610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871df9ce-3cc5-4fe5-b405-e6532f47a49d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.193904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871df9ce-3cc5-4fe5-b405-e6532f47a49d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.194180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871df9ce-3cc5-4fe5-b405-e6532f47a49d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.202633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871df9ce-3cc5-4fe5-b405-e6532f47a49d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:39.204074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871df9ce-3cc5-4fe5-b405-e6532f47a49d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-871df9ce-3cc5-4fe5-b405-e6532f47a49d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:39.216794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:39.217820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:39.219117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:39.219117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:39.219117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:39.219117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:39.219117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:39.219117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:39.219117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:39.219117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:39.219117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 452/896] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:39.219117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871df9ce-3cc5-4fe5-b405-e6532f47a49d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:39.285256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3054d72-cc50-4791-8017-b06c02e28f10 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:39.311209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:39.323336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcff3d74-9b78-47b1-a09b-2832aee59fbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:39.324393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcff3d74-9b78-47b1-a09b-2832aee59fbd None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.324882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcff3d74-9b78-47b1-a09b-2832aee59fbd None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.325357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcff3d74-9b78-47b1-a09b-2832aee59fbd None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.326402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcff3d74-9b78-47b1-a09b-2832aee59fbd None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:39.328086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871df9ce-3cc5-4fe5-b405-e6532f47a49d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:39.328726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcff3d74-9b78-47b1-a09b-2832aee59fbd None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-bcff3d74-9b78-47b1-a09b-2832aee59fbd None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:39.337817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:39.339012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:39.339836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 445/897] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:39 2022] GET /identity/v3/services/object-store => generated 93 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:39.340693 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcff3d74-9b78-47b1-a09b-2832aee59fbd None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:39.366969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f79215-9e76-4f3f-9195-955abc1ee7be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:39.367399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:39.367599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f79215-9e76-4f3f-9195-955abc1ee7be None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.367865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f79215-9e76-4f3f-9195-955abc1ee7be None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.368122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f79215-9e76-4f3f-9195-955abc1ee7be None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.368718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f79215-9e76-4f3f-9195-955abc1ee7be None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:39.370086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f79215-9e76-4f3f-9195-955abc1ee7be None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:39.376357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 453/898] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Sep 13 15:17:39 2022] GET /identity/v3/services => generated 1268 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:17:39.377486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c62247-ca96-4193-a8b9-26fde629c0c6 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.377772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c62247-ca96-4193-a8b9-26fde629c0c6 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.378040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c62247-ca96-4193-a8b9-26fde629c0c6 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.379546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 454/899] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:39.380295 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c62247-ca96-4193-a8b9-26fde629c0c6 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:39.448745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcff3d74-9b78-47b1-a09b-2832aee59fbd None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:39.484534 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab1d0d4-d2ef-4f4e-b09a-205eb9243530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:39.485151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab1d0d4-d2ef-4f4e-b09a-205eb9243530 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.485423 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab1d0d4-d2ef-4f4e-b09a-205eb9243530 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.485693 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab1d0d4-d2ef-4f4e-b09a-205eb9243530 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.486424 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab1d0d4-d2ef-4f4e-b09a-205eb9243530 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:39.486737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab1d0d4-d2ef-4f4e-b09a-205eb9243530 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:39.488535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab1d0d4-d2ef-4f4e-b09a-205eb9243530 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:39.501645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 446/900] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:39.502885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1a87e9-0879-488d-8939-bd3ab4910975 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.503167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1a87e9-0879-488d-8939-bd3ab4910975 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.503438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1a87e9-0879-488d-8939-bd3ab4910975 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.511453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c62247-ca96-4193-a8b9-26fde629c0c6 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:39.523028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-0a1a87e9-0879-488d-8939-bd3ab4910975 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:39.524308 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0a1a87e9-0879-488d-8939-bd3ab4910975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:39.551810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23be37f6-d997-41cf-ab36-6f0c9969218e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:39.552625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23be37f6-d997-41cf-ab36-6f0c9969218e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.552904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23be37f6-d997-41cf-ab36-6f0c9969218e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.553245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23be37f6-d997-41cf-ab36-6f0c9969218e None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.553950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23be37f6-d997-41cf-ab36-6f0c9969218e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:39.554351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23be37f6-d997-41cf-ab36-6f0c9969218e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:39.556299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23be37f6-d997-41cf-ab36-6f0c9969218e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:39.566154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 455/901] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Sep 13 15:17:39 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:39.567125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23be37f6-d997-41cf-ab36-6f0c9969218e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:39.584091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:39.601512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 447/902] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:39 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:39.602533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1a87e9-0879-488d-8939-bd3ab4910975 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:39.618340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:39.704291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23be37f6-d997-41cf-ab36-6f0c9969218e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:39.738895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1a87e9-0879-488d-8939-bd3ab4910975 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:39.752305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b4798a-e41b-46e4-a369-0d111b4524d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:39.753175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b4798a-e41b-46e4-a369-0d111b4524d5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.753850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b4798a-e41b-46e4-a369-0d111b4524d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.754298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b4798a-e41b-46e4-a369-0d111b4524d5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.755768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b4798a-e41b-46e4-a369-0d111b4524d5 None admin] RBAC: Adding query filter params (role.id=523ec7cacd954bfcab0ae561f38f09e9, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=eac26eaa3b1a4c8c849a45ed9ba20301) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:39.756347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b4798a-e41b-46e4-a369-0d111b4524d5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:39.758599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b4798a-e41b-46e4-a369-0d111b4524d5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:39.766770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 456/903] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:39 2022] GET /identity/v3/role_assignments?user.id=eac26eaa3b1a4c8c849a45ed9ba20301&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=523ec7cacd954bfcab0ae561f38f09e9 => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:39.784182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dff007-1905-4d8a-93fd-4ddd410dffcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:39.785106 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dff007-1905-4d8a-93fd-4ddd410dffcc None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:39.785568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dff007-1905-4d8a-93fd-4ddd410dffcc None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:39.785971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dff007-1905-4d8a-93fd-4ddd410dffcc None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:39.786812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dff007-1905-4d8a-93fd-4ddd410dffcc None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:39.787286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dff007-1905-4d8a-93fd-4ddd410dffcc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:39.788878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74dff007-1905-4d8a-93fd-4ddd410dffcc None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:39.795338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 448/904] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Sep 13 15:17:39 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:39.893441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b4798a-e41b-46e4-a369-0d111b4524d5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:39.987719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:40.015939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b4798a-e41b-46e4-a369-0d111b4524d5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:40.052432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5938b8a1-fa5a-403c-8365-39b4b39fc57b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:40.053462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5938b8a1-fa5a-403c-8365-39b4b39fc57b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:40.053968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5938b8a1-fa5a-403c-8365-39b4b39fc57b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:40.054470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5938b8a1-fa5a-403c-8365-39b4b39fc57b None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:40.055257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5938b8a1-fa5a-403c-8365-39b4b39fc57b None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:40.056827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5938b8a1-fa5a-403c-8365-39b4b39fc57b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-5938b8a1-fa5a-403c-8365-39b4b39fc57b None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:40.066305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:40.067361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:17:40.068212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:40.071118 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 457/905] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:17:39 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:40.077741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dff007-1905-4d8a-93fd-4ddd410dffcc None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:40.168178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:40.191870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74dff007-1905-4d8a-93fd-4ddd410dffcc None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:40.227950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9291724c-88aa-4fd1-a766-92ea30660c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:40.228951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9291724c-88aa-4fd1-a766-92ea30660c69 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:40.229430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9291724c-88aa-4fd1-a766-92ea30660c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:40.229901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9291724c-88aa-4fd1-a766-92ea30660c69 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:40.230818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9291724c-88aa-4fd1-a766-92ea30660c69 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:40.231506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9291724c-88aa-4fd1-a766-92ea30660c69 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:40.233123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9291724c-88aa-4fd1-a766-92ea30660c69 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:40.239117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 449/906] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:17:40 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:40.244487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5938b8a1-fa5a-403c-8365-39b4b39fc57b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:40.357054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5938b8a1-fa5a-403c-8365-39b4b39fc57b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:40.391990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28355c12-497c-4185-9acf-b519156a7243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:40.392671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28355c12-497c-4185-9acf-b519156a7243 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:40.392952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28355c12-497c-4185-9acf-b519156a7243 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:40.393227 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28355c12-497c-4185-9acf-b519156a7243 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:40.393907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28355c12-497c-4185-9acf-b519156a7243 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:40.394248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28355c12-497c-4185-9acf-b519156a7243 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:40.395624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28355c12-497c-4185-9acf-b519156a7243 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:40.400855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 458/907] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:17:40 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:40.407657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9291724c-88aa-4fd1-a766-92ea30660c69 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:40.520140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9291724c-88aa-4fd1-a766-92ea30660c69 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:40.555287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933a7e4d-c268-47d2-a6b6-003d3a328d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:40.555905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a7e4d-c268-47d2-a6b6-003d3a328d3c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:40.556182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a7e4d-c268-47d2-a6b6-003d3a328d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:40.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a7e4d-c268-47d2-a6b6-003d3a328d3c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:40.557163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933a7e4d-c268-47d2-a6b6-003d3a328d3c None admin] RBAC: Adding query filter params (interface=public, service_id=1b3a7fd19b174b0888998e5529620090, region_id=RegionOne) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:40.557468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933a7e4d-c268-47d2-a6b6-003d3a328d3c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:40.558878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933a7e4d-c268-47d2-a6b6-003d3a328d3c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:40.564720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 450/908] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Sep 13 15:17:40 2022] GET /identity/v3/endpoints?service_id=1b3a7fd19b174b0888998e5529620090&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:40.570459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28355c12-497c-4185-9acf-b519156a7243 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:40.575438 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:40.693692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28355c12-497c-4185-9acf-b519156a7243 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:40.729652 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472bd259-d8aa-4aff-81b1-7779639925ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:40.730309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472bd259-d8aa-4aff-81b1-7779639925ad None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:40.730589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472bd259-d8aa-4aff-81b1-7779639925ad None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:40.730863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472bd259-d8aa-4aff-81b1-7779639925ad None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:40.731605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472bd259-d8aa-4aff-81b1-7779639925ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:40.733007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472bd259-d8aa-4aff-81b1-7779639925ad None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:40.739653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 459/909] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Sep 13 15:17:40 2022] GET /identity/v3/services => generated 1268 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:17:41.603474 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94834711-3e87-46a5-a591-f5e1775d3ef4 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:41.604052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94834711-3e87-46a5-a591-f5e1775d3ef4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:41.604052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94834711-3e87-46a5-a591-f5e1775d3ef4 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:41.605617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 451/910] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:41.611500 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828d052-1846-490f-8638-926999f27330 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:41.611961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828d052-1846-490f-8638-926999f27330 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:41.612058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b828d052-1846-490f-8638-926999f27330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:41.632611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b828d052-1846-490f-8638-926999f27330 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:41.634216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b828d052-1846-490f-8638-926999f27330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:41.706956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:41.713170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 460/911] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:41 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:41.974342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94834711-3e87-46a5-a591-f5e1775d3ef4 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:41.985702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:41.991407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d79a24-46c3-4ece-993d-7c0a218fd2b6 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:41.991700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d79a24-46c3-4ece-993d-7c0a218fd2b6 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:41.991970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d79a24-46c3-4ece-993d-7c0a218fd2b6 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:41.993575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 461/912] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:42.000513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189b78bd-0d62-4ec3-a3c3-ea7c00abb6f7 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:42.000791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189b78bd-0d62-4ec3-a3c3-ea7c00abb6f7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:42.001065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189b78bd-0d62-4ec3-a3c3-ea7c00abb6f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:42.025003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-189b78bd-0d62-4ec3-a3c3-ea7c00abb6f7 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:42.026600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-189b78bd-0d62-4ec3-a3c3-ea7c00abb6f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:42.092739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94834711-3e87-46a5-a591-f5e1775d3ef4 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:42.100052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 462/913] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:41 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:42.101543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077c892c-5b10-4068-bf61-79ad516fef75 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:42.102160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077c892c-5b10-4068-bf61-79ad516fef75 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:42.102602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077c892c-5b10-4068-bf61-79ad516fef75 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:42.103971 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 463/914] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:42.109509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4979650-8b7a-4f19-8ae2-4b2a2dca415b None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:42.109788 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4979650-8b7a-4f19-8ae2-4b2a2dca415b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:42.110056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4979650-8b7a-4f19-8ae2-4b2a2dca415b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:42.132232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c4979650-8b7a-4f19-8ae2-4b2a2dca415b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:42.133412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c4979650-8b7a-4f19-8ae2-4b2a2dca415b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:42.135506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3498b27d-4a64-4ca8-ad75-087c3cd1855e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:42.136514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3498b27d-4a64-4ca8-ad75-087c3cd1855e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:42.136851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3498b27d-4a64-4ca8-ad75-087c3cd1855e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:42.137255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3498b27d-4a64-4ca8-ad75-087c3cd1855e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:42.138541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3498b27d-4a64-4ca8-ad75-087c3cd1855e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:42.139989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3498b27d-4a64-4ca8-ad75-087c3cd1855e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-3498b27d-4a64-4ca8-ad75-087c3cd1855e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:42.149319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:42.150307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:42.156381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 452/915] 10.222.0.16 () {56 vars in 1132 bytes} [Tue Sep 13 15:17:41 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:42.157178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3498b27d-4a64-4ca8-ad75-087c3cd1855e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:42.223125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 464/916] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:42 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:42.249583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:42.281761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3498b27d-4a64-4ca8-ad75-087c3cd1855e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:42.316456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3debda8-6b92-46b1-88f8-ea02b252d722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:42.317488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3debda8-6b92-46b1-88f8-ea02b252d722 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:42.317958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3debda8-6b92-46b1-88f8-ea02b252d722 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:42.318444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3debda8-6b92-46b1-88f8-ea02b252d722 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:42.319287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3debda8-6b92-46b1-88f8-ea02b252d722 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:42.319776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3debda8-6b92-46b1-88f8-ea02b252d722 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:42.321330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3debda8-6b92-46b1-88f8-ea02b252d722 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:42.326611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 453/917] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:42 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:42.332499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4979650-8b7a-4f19-8ae2-4b2a2dca415b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:42.389717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3debda8-6b92-46b1-88f8-ea02b252d722 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:42.396300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:42.440936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4979650-8b7a-4f19-8ae2-4b2a2dca415b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:42.476794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991c53af-e895-428e-9267-c6b105031548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:42.477295 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991c53af-e895-428e-9267-c6b105031548 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:42.477570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991c53af-e895-428e-9267-c6b105031548 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:42.477838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991c53af-e895-428e-9267-c6b105031548 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:42.478507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991c53af-e895-428e-9267-c6b105031548 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:42.478806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991c53af-e895-428e-9267-c6b105031548 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:42.480168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991c53af-e895-428e-9267-c6b105031548 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:42.489957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 465/918] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:42 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:42.493542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991c53af-e895-428e-9267-c6b105031548 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:42.541306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3debda8-6b92-46b1-88f8-ea02b252d722 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:42.575531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4eaba-5689-4b19-97fc-e21e073c643a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:42.576172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4eaba-5689-4b19-97fc-e21e073c643a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:42.576450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4eaba-5689-4b19-97fc-e21e073c643a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:42.576727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d4eaba-5689-4b19-97fc-e21e073c643a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:42.578728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:42.588340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d4eaba-5689-4b19-97fc-e21e073c643a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-c0d4eaba-5689-4b19-97fc-e21e073c643a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:42.593610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:42.594595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:17:42.595471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:42.599710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 454/919] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Sep 13 15:17:42 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:42.600404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4eaba-5689-4b19-97fc-e21e073c643a None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:42.604714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991c53af-e895-428e-9267-c6b105031548 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:42.630392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:42.643280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ec9460-e590-4b0d-b734-c43cf282a194 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:42.643902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec9460-e590-4b0d-b734-c43cf282a194 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:42.644176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec9460-e590-4b0d-b734-c43cf282a194 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:42.644446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec9460-e590-4b0d-b734-c43cf282a194 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:42.645132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ec9460-e590-4b0d-b734-c43cf282a194 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:42.646509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ec9460-e590-4b0d-b734-c43cf282a194 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:42.670052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 466/920] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:17:42 2022] POST /identity/v3/endpoints => generated 347 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:17:42.671220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ec9460-e590-4b0d-b734-c43cf282a194 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:42.741523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d4eaba-5689-4b19-97fc-e21e073c643a None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:42.787017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a566f565-b43b-4c9d-889c-526db49d58ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:42.788130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a566f565-b43b-4c9d-889c-526db49d58ac None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:42.788628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a566f565-b43b-4c9d-889c-526db49d58ac None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:42.789133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a566f565-b43b-4c9d-889c-526db49d58ac None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:42.790026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a566f565-b43b-4c9d-889c-526db49d58ac None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:42.791864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a566f565-b43b-4c9d-889c-526db49d58ac None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:42.802424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ec9460-e590-4b0d-b734-c43cf282a194 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:42.822174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 455/921] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:17:42 2022] POST /identity/v3/services => generated 241 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:17:42.876876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885abf0d-7c8c-4f32-8d6b-a410cf168b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:42.877821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885abf0d-7c8c-4f32-8d6b-a410cf168b22 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:42.878275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885abf0d-7c8c-4f32-8d6b-a410cf168b22 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:42.878722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885abf0d-7c8c-4f32-8d6b-a410cf168b22 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:42.879565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885abf0d-7c8c-4f32-8d6b-a410cf168b22 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:42.880021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885abf0d-7c8c-4f32-8d6b-a410cf168b22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:42.881709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885abf0d-7c8c-4f32-8d6b-a410cf168b22 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:42.893086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 467/922] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Sep 13 15:17:42 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:42.899914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a566f565-b43b-4c9d-889c-526db49d58ac None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:42.960491 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:43.004646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e58eebd-0bb3-4446-8b83-29bbfd4573af None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:43.005026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e58eebd-0bb3-4446-8b83-29bbfd4573af None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:43.005219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e58eebd-0bb3-4446-8b83-29bbfd4573af None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:43.008283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 468/923] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:43.013579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395d2121-0d4f-42e1-aa62-55716ec81d29 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:43.013853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395d2121-0d4f-42e1-aa62-55716ec81d29 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:43.014159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395d2121-0d4f-42e1-aa62-55716ec81d29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:43.030421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a566f565-b43b-4c9d-889c-526db49d58ac None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:43.034820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-395d2121-0d4f-42e1-aa62-55716ec81d29 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:43.036010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-395d2121-0d4f-42e1-aa62-55716ec81d29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:43.068352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:43.069262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa147bbe-fcbe-4b48-9a7c-6660e6055090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:43.069745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa147bbe-fcbe-4b48-9a7c-6660e6055090 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:43.070001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa147bbe-fcbe-4b48-9a7c-6660e6055090 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:43.070351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa147bbe-fcbe-4b48-9a7c-6660e6055090 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:43.085808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa147bbe-fcbe-4b48-9a7c-6660e6055090 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:43.087414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa147bbe-fcbe-4b48-9a7c-6660e6055090 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-aa147bbe-fcbe-4b48-9a7c-6660e6055090 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:43.103808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:43.104810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:43.105694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:43.105694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:43.105694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:43.105694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 456/924] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:43.111205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa147bbe-fcbe-4b48-9a7c-6660e6055090 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:43.114380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 469/925] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:43 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:43.217925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa147bbe-fcbe-4b48-9a7c-6660e6055090 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:43.219503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:43.254370 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd33211-1180-4744-8a10-2be788f636a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:43.254988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd33211-1180-4744-8a10-2be788f636a6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:43.255270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd33211-1180-4744-8a10-2be788f636a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:43.255542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd33211-1180-4744-8a10-2be788f636a6 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:43.256188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd33211-1180-4744-8a10-2be788f636a6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:43.256485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd33211-1180-4744-8a10-2be788f636a6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:43.257972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd33211-1180-4744-8a10-2be788f636a6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:43.269887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 457/926] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:43.275820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395d2121-0d4f-42e1-aa62-55716ec81d29 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:43.375614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395d2121-0d4f-42e1-aa62-55716ec81d29 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:43.409616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f06ce1-d8d2-42e5-bfd2-cbada6e09c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:43.410324 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f06ce1-d8d2-42e5-bfd2-cbada6e09c3b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:43.410600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f06ce1-d8d2-42e5-bfd2-cbada6e09c3b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:43.410913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f06ce1-d8d2-42e5-bfd2-cbada6e09c3b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:43.412735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd33211-1180-4744-8a10-2be788f636a6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:43.419053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f06ce1-d8d2-42e5-bfd2-cbada6e09c3b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:43.420748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f06ce1-d8d2-42e5-bfd2-cbada6e09c3b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-d1f06ce1-d8d2-42e5-bfd2-cbada6e09c3b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:43.433600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:43.434795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:43.435948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:43.435948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:43.435948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:43.435948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:43.435948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:43.435948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:43.435948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:43.435948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:43.435948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:43.435948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:43.438211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 470/927] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Sep 13 15:17:43 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:43.445820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f06ce1-d8d2-42e5-bfd2-cbada6e09c3b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:43.478663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:43.529965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd33211-1180-4744-8a10-2be788f636a6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:43.567183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab68e2e-0345-4f8d-99a5-3cacd47bea60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:43.567914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab68e2e-0345-4f8d-99a5-3cacd47bea60 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:43.568285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab68e2e-0345-4f8d-99a5-3cacd47bea60 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:43.568650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab68e2e-0345-4f8d-99a5-3cacd47bea60 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:43.569324 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab68e2e-0345-4f8d-99a5-3cacd47bea60 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:43.570820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab68e2e-0345-4f8d-99a5-3cacd47bea60 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:43.575307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f06ce1-d8d2-42e5-bfd2-cbada6e09c3b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-fab68e2e-0345-4f8d-99a5-3cacd47bea60 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:43.580199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:43.581398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 13 15:17:43.582468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:43.589101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 458/928] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:17:43 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:43.590485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab68e2e-0345-4f8d-99a5-3cacd47bea60 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:43.620863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:43.628020 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fb1b7a-15d1-4bc1-bd44-f9d9fe8765a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:43.628887 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fb1b7a-15d1-4bc1-bd44-f9d9fe8765a4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:43.629593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fb1b7a-15d1-4bc1-bd44-f9d9fe8765a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:43.630013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fb1b7a-15d1-4bc1-bd44-f9d9fe8765a4 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:43.630944 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fb1b7a-15d1-4bc1-bd44-f9d9fe8765a4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:43.631817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fb1b7a-15d1-4bc1-bd44-f9d9fe8765a4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:43.634424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fb1b7a-15d1-4bc1-bd44-f9d9fe8765a4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:43.657376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 471/929] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Sep 13 15:17:43 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:43.662936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fb1b7a-15d1-4bc1-bd44-f9d9fe8765a4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:43.708913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab68e2e-0345-4f8d-99a5-3cacd47bea60 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:43.748042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11754536-b55d-4691-b8da-e4e91379a2ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:43.748691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11754536-b55d-4691-b8da-e4e91379a2ce None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:43.748945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11754536-b55d-4691-b8da-e4e91379a2ce None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:43.749234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11754536-b55d-4691-b8da-e4e91379a2ce None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:43.749979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11754536-b55d-4691-b8da-e4e91379a2ce None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:43.750309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11754536-b55d-4691-b8da-e4e91379a2ce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:43.751793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11754536-b55d-4691-b8da-e4e91379a2ce None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:43.761832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 459/930] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:17:43 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:43.763316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11754536-b55d-4691-b8da-e4e91379a2ce None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:43.781018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:43.802028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fb1b7a-15d1-4bc1-bd44-f9d9fe8765a4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:43.843207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d579c93e-3250-476c-9643-45d85cda1c80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:43.843898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d579c93e-3250-476c-9643-45d85cda1c80 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:43.844177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d579c93e-3250-476c-9643-45d85cda1c80 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:43.844451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d579c93e-3250-476c-9643-45d85cda1c80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:43.856365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d579c93e-3250-476c-9643-45d85cda1c80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:43.857876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d579c93e-3250-476c-9643-45d85cda1c80 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-d579c93e-3250-476c-9643-45d85cda1c80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:43.874649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:43.875852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:43.877115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:43.877115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:43.877115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:43.877115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 472/931] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:43.880815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:43.881899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d579c93e-3250-476c-9643-45d85cda1c80 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:43.894207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11754536-b55d-4691-b8da-e4e91379a2ce None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:43.947274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cae1660-0877-4c03-aefe-58f0b6c6d9c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:43.948960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cae1660-0877-4c03-aefe-58f0b6c6d9c3 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:43.949422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cae1660-0877-4c03-aefe-58f0b6c6d9c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:43.950033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cae1660-0877-4c03-aefe-58f0b6c6d9c3 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:43.951635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cae1660-0877-4c03-aefe-58f0b6c6d9c3 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:43.952629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cae1660-0877-4c03-aefe-58f0b6c6d9c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:43.954725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cae1660-0877-4c03-aefe-58f0b6c6d9c3 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:43.966343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 460/932] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Sep 13 15:17:43 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:43.969324 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cae1660-0877-4c03-aefe-58f0b6c6d9c3 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:44.026238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d579c93e-3250-476c-9643-45d85cda1c80 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:44.068604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dadd2d0-c2ff-4463-b075-f8e5146642e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:44.069219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadd2d0-c2ff-4463-b075-f8e5146642e8 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:44.069496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadd2d0-c2ff-4463-b075-f8e5146642e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:44.069761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadd2d0-c2ff-4463-b075-f8e5146642e8 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:44.070435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dadd2d0-c2ff-4463-b075-f8e5146642e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:44.070734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dadd2d0-c2ff-4463-b075-f8e5146642e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:44.072106 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dadd2d0-c2ff-4463-b075-f8e5146642e8 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:44.075596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:44.089415 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 473/933] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:44.095085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dadd2d0-c2ff-4463-b075-f8e5146642e8 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:44.110524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cae1660-0877-4c03-aefe-58f0b6c6d9c3 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:44.153037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dc8554-a56f-4822-92e4-fc54d377ab98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:44.153893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dc8554-a56f-4822-92e4-fc54d377ab98 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:44.154351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dc8554-a56f-4822-92e4-fc54d377ab98 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:44.154754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dc8554-a56f-4822-92e4-fc54d377ab98 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:44.155560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1dc8554-a56f-4822-92e4-fc54d377ab98 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:44.157112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1dc8554-a56f-4822-92e4-fc54d377ab98 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:44.184460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 461/934] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:17:43 2022] POST /identity/v3/endpoints => generated 346 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:17:44.233518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dadd2d0-c2ff-4463-b075-f8e5146642e8 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:44.281728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adf08ce-7ab8-4c2d-bb4e-a593e6c32ada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:44.282744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adf08ce-7ab8-4c2d-bb4e-a593e6c32ada None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:44.283200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adf08ce-7ab8-4c2d-bb4e-a593e6c32ada None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:44.283610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adf08ce-7ab8-4c2d-bb4e-a593e6c32ada None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:44.292386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adf08ce-7ab8-4c2d-bb4e-a593e6c32ada None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:44.294344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adf08ce-7ab8-4c2d-bb4e-a593e6c32ada None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-7adf08ce-7ab8-4c2d-bb4e-a593e6c32ada None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:44.308171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:44.309288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:44.310518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:44.310518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:44.310518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:44.310518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:44.310518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:44.310518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:44.310518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:44.310518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:44.312814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 474/935] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:44.318948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dc8554-a56f-4822-92e4-fc54d377ab98 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:44.335361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:44.431635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dc8554-a56f-4822-92e4-fc54d377ab98 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:44.467147 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb2e461-94c7-4164-9dcb-001048270484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:44.467762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2e461-94c7-4164-9dcb-001048270484 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:44.468040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2e461-94c7-4164-9dcb-001048270484 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:44.468303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2e461-94c7-4164-9dcb-001048270484 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:44.468987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb2e461-94c7-4164-9dcb-001048270484 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:44.469286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb2e461-94c7-4164-9dcb-001048270484 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:44.470814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb2e461-94c7-4164-9dcb-001048270484 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:44.483341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 462/936] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:44.489844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adf08ce-7ab8-4c2d-bb4e-a593e6c32ada None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:44.545747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:44.595410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adf08ce-7ab8-4c2d-bb4e-a593e6c32ada None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:44.630749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e020b3c0-b329-4d52-8c46-6222238bb410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:44.631527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e020b3c0-b329-4d52-8c46-6222238bb410 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:44.632067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e020b3c0-b329-4d52-8c46-6222238bb410 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:44.632165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e020b3c0-b329-4d52-8c46-6222238bb410 None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/eac26eaa3b1a4c8c849a45ed9ba20301/roles/523ec7cacd954bfcab0ae561f38f09e9` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:44.634422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e020b3c0-b329-4d52-8c46-6222238bb410 None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=eac26eaa3b1a4c8c849a45ed9ba20301, role_id=523ec7cacd954bfcab0ae561f38f09e9)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:44.635901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e020b3c0-b329-4d52-8c46-6222238bb410 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:44.657607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 475/937] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:17:44 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/eac26eaa3b1a4c8c849a45ed9ba20301/roles/523ec7cacd954bfcab0ae561f38f09e9 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:17:45.062327 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3c8a74-7b6a-4d93-85af-6b993dc326c1 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:45.062889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3c8a74-7b6a-4d93-85af-6b993dc326c1 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:45.062889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3c8a74-7b6a-4d93-85af-6b993dc326c1 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:45.064416 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 463/938] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:45.069905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d910c97-8479-4198-b132-c56d0c094ee1 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:45.070212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d910c97-8479-4198-b132-c56d0c094ee1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:45.070481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d910c97-8479-4198-b132-c56d0c094ee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:45.089443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0d910c97-8479-4198-b132-c56d0c094ee1 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:45.090958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0d910c97-8479-4198-b132-c56d0c094ee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:45.164480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 476/939] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:45.246429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e86a57-1340-43b6-921e-8eadfa54baf6 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:45.246869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e86a57-1340-43b6-921e-8eadfa54baf6 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:45.247457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e86a57-1340-43b6-921e-8eadfa54baf6 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:45.248609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 464/940] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:45.253851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b60f602-649d-41af-bcb2-6c8adf7150e3 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:45.254152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b60f602-649d-41af-bcb2-6c8adf7150e3 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:45.254427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b60f602-649d-41af-bcb2-6c8adf7150e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:45.272929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8b60f602-649d-41af-bcb2-6c8adf7150e3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:45.274097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8b60f602-649d-41af-bcb2-6c8adf7150e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:45.310197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:45.343675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 477/941] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:45.436742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e86a57-1340-43b6-921e-8eadfa54baf6 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:45.511513 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:45.545432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e86a57-1340-43b6-921e-8eadfa54baf6 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:45.585053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5820a1a5-335f-41a1-aede-79b9d0d10614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:45.585692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5820a1a5-335f-41a1-aede-79b9d0d10614 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:45.586015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5820a1a5-335f-41a1-aede-79b9d0d10614 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:45.586313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5820a1a5-335f-41a1-aede-79b9d0d10614 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:45.586894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5820a1a5-335f-41a1-aede-79b9d0d10614 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:45.588316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5820a1a5-335f-41a1-aede-79b9d0d10614 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-5820a1a5-335f-41a1-aede-79b9d0d10614 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:45.598179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:45.599419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:45.600429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 465/942] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:45 2022] GET /identity/v3/services/compute => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:45.607504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b60f602-649d-41af-bcb2-6c8adf7150e3 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:45.635349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5820a1a5-335f-41a1-aede-79b9d0d10614 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:45.720686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b60f602-649d-41af-bcb2-6c8adf7150e3 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:45.749614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5820a1a5-335f-41a1-aede-79b9d0d10614 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:45.757633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d83b6bb-cb2b-4ddd-8007-3d00c7190f37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:45.758476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83b6bb-cb2b-4ddd-8007-3d00c7190f37 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:45.758757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83b6bb-cb2b-4ddd-8007-3d00c7190f37 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:45.759018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83b6bb-cb2b-4ddd-8007-3d00c7190f37 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:45.759646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d83b6bb-cb2b-4ddd-8007-3d00c7190f37 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:45.759948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d83b6bb-cb2b-4ddd-8007-3d00c7190f37 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:45.761320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d83b6bb-cb2b-4ddd-8007-3d00c7190f37 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:45.766491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 478/943] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:45 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:45.771972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d83b6bb-cb2b-4ddd-8007-3d00c7190f37 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:45.790541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94259731-50e0-4016-a038-0fdb149b2cef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:45.791265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94259731-50e0-4016-a038-0fdb149b2cef None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:45.791464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94259731-50e0-4016-a038-0fdb149b2cef None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:45.791737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94259731-50e0-4016-a038-0fdb149b2cef None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:45.792367 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94259731-50e0-4016-a038-0fdb149b2cef None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:45.793920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94259731-50e0-4016-a038-0fdb149b2cef None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:45.795701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-94259731-50e0-4016-a038-0fdb149b2cef None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:45.803191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:45.804224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:45.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:45.809655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 466/944] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:45 2022] GET /identity/v3/services/object-store => generated 93 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:45.810708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94259731-50e0-4016-a038-0fdb149b2cef None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:45.821369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:45.900080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d83b6bb-cb2b-4ddd-8007-3d00c7190f37 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:45.919971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94259731-50e0-4016-a038-0fdb149b2cef None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:45.938024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ce1c32-37e0-4dc6-be08-bc171d867c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:45.938675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ce1c32-37e0-4dc6-be08-bc171d867c1b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:45.938947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ce1c32-37e0-4dc6-be08-bc171d867c1b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:45.939219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ce1c32-37e0-4dc6-be08-bc171d867c1b None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:45.939847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ce1c32-37e0-4dc6-be08-bc171d867c1b None admin] RBAC: Adding query filter params (type=compute) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:45.940150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ce1c32-37e0-4dc6-be08-bc171d867c1b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:45.941502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ce1c32-37e0-4dc6-be08-bc171d867c1b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:45.948666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 479/945] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:45 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:45.964380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4d08cd-1fb7-4551-b227-85a374ceca6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:45.965074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4d08cd-1fb7-4551-b227-85a374ceca6f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:45.965385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4d08cd-1fb7-4551-b227-85a374ceca6f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:45.965653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4d08cd-1fb7-4551-b227-85a374ceca6f None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:45.966623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4d08cd-1fb7-4551-b227-85a374ceca6f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:45.966919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4d08cd-1fb7-4551-b227-85a374ceca6f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:45.968354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4d08cd-1fb7-4551-b227-85a374ceca6f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:45.974430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 467/946] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Sep 13 15:17:45 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:45.983685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ce1c32-37e0-4dc6-be08-bc171d867c1b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:46.091840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:46.098923 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ce1c32-37e0-4dc6-be08-bc171d867c1b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:46.138976 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238132c5-2cd0-44e2-a2eb-3409b502b764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:46.139607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238132c5-2cd0-44e2-a2eb-3409b502b764 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:46.139882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238132c5-2cd0-44e2-a2eb-3409b502b764 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:46.140152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238132c5-2cd0-44e2-a2eb-3409b502b764 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:46.140794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238132c5-2cd0-44e2-a2eb-3409b502b764 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:46.141095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238132c5-2cd0-44e2-a2eb-3409b502b764 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:46.142577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238132c5-2cd0-44e2-a2eb-3409b502b764 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:46.148924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 480/947] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Sep 13 15:17:45 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:46.154569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4d08cd-1fb7-4551-b227-85a374ceca6f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:46.228282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:46.269574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4d08cd-1fb7-4551-b227-85a374ceca6f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:46.305328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7d6bed-5fc4-45c3-bc71-93bd6e1cbfe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:46.305948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d6bed-5fc4-45c3-bc71-93bd6e1cbfe6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:46.306255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d6bed-5fc4-45c3-bc71-93bd6e1cbfe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:46.306584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d6bed-5fc4-45c3-bc71-93bd6e1cbfe6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:46.307289 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7d6bed-5fc4-45c3-bc71-93bd6e1cbfe6 None admin] RBAC: Adding query filter params (interface=public, service_id=21f856fac94a421eb9139fe3a536bb8a, region_id=RegionOne) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:46.307706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7d6bed-5fc4-45c3-bc71-93bd6e1cbfe6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:46.309186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7d6bed-5fc4-45c3-bc71-93bd6e1cbfe6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:46.315200 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 468/948] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Sep 13 15:17:46 2022] GET /identity/v3/endpoints?service_id=21f856fac94a421eb9139fe3a536bb8a&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:46.321177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238132c5-2cd0-44e2-a2eb-3409b502b764 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:46.423958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238132c5-2cd0-44e2-a2eb-3409b502b764 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:46.461088 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee99fa4a-f9e0-49ae-9b6c-78c2d6e934a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:46.461728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee99fa4a-f9e0-49ae-9b6c-78c2d6e934a1 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:46.462003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee99fa4a-f9e0-49ae-9b6c-78c2d6e934a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:46.462299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee99fa4a-f9e0-49ae-9b6c-78c2d6e934a1 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:46.462930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee99fa4a-f9e0-49ae-9b6c-78c2d6e934a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:46.464338 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee99fa4a-f9e0-49ae-9b6c-78c2d6e934a1 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:46.471900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 481/949] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Sep 13 15:17:46 2022] GET /identity/v3/services => generated 1497 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:17:46.579963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64be014-0252-499c-8557-be03e61fbf65 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:46.580301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64be014-0252-499c-8557-be03e61fbf65 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:46.580614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64be014-0252-499c-8557-be03e61fbf65 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:46.582311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 469/950] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:46.588143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa26955e-01a8-4b22-a54c-b4094e6be148 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:46.588422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa26955e-01a8-4b22-a54c-b4094e6be148 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:46.588697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa26955e-01a8-4b22-a54c-b4094e6be148 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:46.609055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-aa26955e-01a8-4b22-a54c-b4094e6be148 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:46.610342 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-aa26955e-01a8-4b22-a54c-b4094e6be148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:46.636879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:46.684508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 482/951] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:46 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:46.933475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe3a0c1-99c2-4fd3-a1c8-2eb207a2b021 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:46.933816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe3a0c1-99c2-4fd3-a1c8-2eb207a2b021 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:46.934433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe3a0c1-99c2-4fd3-a1c8-2eb207a2b021 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:46.936054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 470/952] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:46.941439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90e3c66-7ff7-4291-88b2-e9c6244c71c7 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:46.941901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90e3c66-7ff7-4291-88b2-e9c6244c71c7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:46.941976 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90e3c66-7ff7-4291-88b2-e9c6244c71c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:46.954336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe3a0c1-99c2-4fd3-a1c8-2eb207a2b021 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:46.961700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b90e3c66-7ff7-4291-88b2-e9c6244c71c7 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:46.962974 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b90e3c66-7ff7-4291-88b2-e9c6244c71c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:47.050126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 483/953] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:46 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:47.080530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe3a0c1-99c2-4fd3-a1c8-2eb207a2b021 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:47.107066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:47.116344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:47.116971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:47.117258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:47.117536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:47.118217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:47.119678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:47.122465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: WARNING keystone.server.flask.common [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 13 15:17:47.145767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.sql.core [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 13 15:17:47.145767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 13 15:17:47.145767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [parameters: {'id': '4696f948feef414094dcb0aee2ed073e', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 13 15:17:47.145767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81454) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4696f948feef414094dcb0aee2ed073e.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4696f948feef414094dcb0aee2ed073e. Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:17:47.163181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application result.read() Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 13 15:17:47.164457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application yield resource Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.session.rollback() Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.force_reraise() Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise self.value Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application session.commit() Sep 13 15:17:47.165387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._prepare_impl() Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.session.flush() Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._flush(objects) Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application compat.raise_( Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application flush_context.execute() Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 13 15:17:47.167095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rec.execute(self) Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 13 15:17:47.169162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application result.read() Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:17:47.170957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [parameters: {'id': '4696f948feef414094dcb0aee2ed073e', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:47.171867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:47.172697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:47.173649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:47.173649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 13 15:17:47.173649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 13 15:17:47.173649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:47.173649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:47.173649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 13 15:17:47.173649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 13 15:17:47.173649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 4696f948feef414094dcb0aee2ed073e. Sep 13 15:17:47.173649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:47.173649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 471/954] 10.222.0.16 () {60 vars in 1140 bytes} [Tue Sep 13 15:17:46 2022] POST /identity/v3/projects => generated 262 bytes in 211 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:47.175992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90e3c66-7ff7-4291-88b2-e9c6244c71c7 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:47.267880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:47.280263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90e3c66-7ff7-4291-88b2-e9c6244c71c7 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:47.318121 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0ac1ce-aecf-4219-ae19-c72a09787865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:47.319072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0ac1ce-aecf-4219-ae19-c72a09787865 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:47.319494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0ac1ce-aecf-4219-ae19-c72a09787865 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:47.319900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0ac1ce-aecf-4219-ae19-c72a09787865 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:47.327733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0ac1ce-aecf-4219-ae19-c72a09787865 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:47.329350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0ac1ce-aecf-4219-ae19-c72a09787865 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-bb0ac1ce-aecf-4219-ae19-c72a09787865 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:47.341505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:47.342686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:47.343828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:47.343828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:47.343828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:47.343828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:47.343828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:47.343828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:47.343828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:47.343828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:47.345910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 484/955] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:47.351849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:47.360108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0ac1ce-aecf-4219-ae19-c72a09787865 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:47.460348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0ac1ce-aecf-4219-ae19-c72a09787865 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:47.461446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2207b6b4-f7a9-4d30-b3cd-27599ac30940 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:47.494007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8855b07c-c3e3-4ac3-a02c-3cc49989e1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:47.494835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8855b07c-c3e3-4ac3-a02c-3cc49989e1db None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:47.495209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8855b07c-c3e3-4ac3-a02c-3cc49989e1db None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:47.495572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8855b07c-c3e3-4ac3-a02c-3cc49989e1db None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:47.499573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa206c08-0488-472f-a585-043b8813e491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:47.500196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa206c08-0488-472f-a585-043b8813e491 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:47.500529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa206c08-0488-472f-a585-043b8813e491 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:47.500806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa206c08-0488-472f-a585-043b8813e491 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:47.501469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa206c08-0488-472f-a585-043b8813e491 None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:47.501772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa206c08-0488-472f-a585-043b8813e491 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:47.503553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa206c08-0488-472f-a585-043b8813e491 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:47.508104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8855b07c-c3e3-4ac3-a02c-3cc49989e1db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-8855b07c-c3e3-4ac3-a02c-3cc49989e1db None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:47.512935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:47.513954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:17:47.514877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:47.521492 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 485/956] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Sep 13 15:17:47 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:47.522374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8855b07c-c3e3-4ac3-a02c-3cc49989e1db None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:47.530259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 472/957] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:47 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:47.566163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:47.638585 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8855b07c-c3e3-4ac3-a02c-3cc49989e1db None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:47.679456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ef9f9d-e498-41de-b6f0-a13a3fcc0725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:47.680206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ef9f9d-e498-41de-b6f0-a13a3fcc0725 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:47.680481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ef9f9d-e498-41de-b6f0-a13a3fcc0725 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:47.680769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ef9f9d-e498-41de-b6f0-a13a3fcc0725 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:47.681428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ef9f9d-e498-41de-b6f0-a13a3fcc0725 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:47.681728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ef9f9d-e498-41de-b6f0-a13a3fcc0725 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:47.683440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ef9f9d-e498-41de-b6f0-a13a3fcc0725 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:47.693192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 486/958] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Sep 13 15:17:47 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:47.700208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa206c08-0488-472f-a585-043b8813e491 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:47.748473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:47.810760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa206c08-0488-472f-a585-043b8813e491 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:47.845872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644e4e80-baba-47c8-8634-3ac054b91bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:47.846542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644e4e80-baba-47c8-8634-3ac054b91bb8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:47.846816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644e4e80-baba-47c8-8634-3ac054b91bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:47.847077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644e4e80-baba-47c8-8634-3ac054b91bb8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:47.857943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644e4e80-baba-47c8-8634-3ac054b91bb8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:47.859444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644e4e80-baba-47c8-8634-3ac054b91bb8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-644e4e80-baba-47c8-8634-3ac054b91bb8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:47.877918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:47.878987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:47.880635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:47.880635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:47.880635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:47.880635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 473/959] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:47.885371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ef9f9d-e498-41de-b6f0-a13a3fcc0725 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:47.959143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:47.991021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ef9f9d-e498-41de-b6f0-a13a3fcc0725 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:48.025020 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68c7da7-206d-4e59-8fb3-1d9ab1b08832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:48.025629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68c7da7-206d-4e59-8fb3-1d9ab1b08832 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:48.025901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68c7da7-206d-4e59-8fb3-1d9ab1b08832 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:48.026191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68c7da7-206d-4e59-8fb3-1d9ab1b08832 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:48.026824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68c7da7-206d-4e59-8fb3-1d9ab1b08832 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:48.027123 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68c7da7-206d-4e59-8fb3-1d9ab1b08832 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:48.028567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68c7da7-206d-4e59-8fb3-1d9ab1b08832 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:48.040936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 487/960] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:48.046796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644e4e80-baba-47c8-8634-3ac054b91bb8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:48.146727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:48.156435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644e4e80-baba-47c8-8634-3ac054b91bb8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:48.193749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b5fc5e-c47b-4c7b-acc1-bd37cc23fbce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:48.194416 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b5fc5e-c47b-4c7b-acc1-bd37cc23fbce None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:48.194674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b5fc5e-c47b-4c7b-acc1-bd37cc23fbce None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:48.194940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b5fc5e-c47b-4c7b-acc1-bd37cc23fbce None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:48.202985 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b5fc5e-c47b-4c7b-acc1-bd37cc23fbce None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:48.204398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b5fc5e-c47b-4c7b-acc1-bd37cc23fbce None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-d6b5fc5e-c47b-4c7b-acc1-bd37cc23fbce None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:48.215778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:48.216798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:48.218490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 474/961] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Sep 13 15:17:48 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:48.223019 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68c7da7-206d-4e59-8fb3-1d9ab1b08832 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:48.259803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22acd7d2-277a-4b5d-8489-5aa16ff49f92 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:48.260189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22acd7d2-277a-4b5d-8489-5aa16ff49f92 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:48.260351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22acd7d2-277a-4b5d-8489-5aa16ff49f92 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:48.262041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 475/962] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:48.267237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00472d46-c923-4793-aa72-f4cede7ae07f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:48.267515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00472d46-c923-4793-aa72-f4cede7ae07f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:48.267786 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00472d46-c923-4793-aa72-f4cede7ae07f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:48.289097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-00472d46-c923-4793-aa72-f4cede7ae07f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:48.290914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-00472d46-c923-4793-aa72-f4cede7ae07f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:48.328829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68c7da7-206d-4e59-8fb3-1d9ab1b08832 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:48.360003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:48.367339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af699ee2-9dec-471e-b0bf-83e679ae8f03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:48.368029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af699ee2-9dec-471e-b0bf-83e679ae8f03 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:48.368361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af699ee2-9dec-471e-b0bf-83e679ae8f03 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:48.368633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af699ee2-9dec-471e-b0bf-83e679ae8f03 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:48.369491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af699ee2-9dec-471e-b0bf-83e679ae8f03 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:48.369659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af699ee2-9dec-471e-b0bf-83e679ae8f03 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:48.371119 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af699ee2-9dec-471e-b0bf-83e679ae8f03 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:48.378199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 476/963] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:48.395208 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 488/964] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Sep 13 15:17:48 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:48.400994 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00472d46-c923-4793-aa72-f4cede7ae07f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:48.494201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:48.513328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00472d46-c923-4793-aa72-f4cede7ae07f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:48.550272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ccc52-35c1-4ebc-b1c4-a3bd2b596ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:48.551236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ccc52-35c1-4ebc-b1c4-a3bd2b596ec9 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:48.551691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ccc52-35c1-4ebc-b1c4-a3bd2b596ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:48.552167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ccc52-35c1-4ebc-b1c4-a3bd2b596ec9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:48.631530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2ccc52-35c1-4ebc-b1c4-a3bd2b596ec9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:48.633361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2ccc52-35c1-4ebc-b1c4-a3bd2b596ec9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:48.669640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af699ee2-9dec-471e-b0bf-83e679ae8f03 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:48.725584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:48.748888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 477/965] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:48 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:17:48.748888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c250a03e-49c1-4904-8edc-9280ccc35666 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:48.748888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c250a03e-49c1-4904-8edc-9280ccc35666 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:48.748888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c250a03e-49c1-4904-8edc-9280ccc35666 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:48.748888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 478/966] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:48.750589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c250a03e-49c1-4904-8edc-9280ccc35666 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:48.785799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af699ee2-9dec-471e-b0bf-83e679ae8f03 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:48.822598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f1a989-0ae0-4601-a8d9-4e7bc7e6ead4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:48.823223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f1a989-0ae0-4601-a8d9-4e7bc7e6ead4 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:48.823501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f1a989-0ae0-4601-a8d9-4e7bc7e6ead4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:48.823770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f1a989-0ae0-4601-a8d9-4e7bc7e6ead4 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:48.824477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f1a989-0ae0-4601-a8d9-4e7bc7e6ead4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:48.826736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f1a989-0ae0-4601-a8d9-4e7bc7e6ead4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:48.844342 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 489/967] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:17:48 2022] POST /identity/v3/services => generated 242 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:17:48.845828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb8feb4-5fc4-4be6-91b4-03e21a88a636 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:48.846380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb8feb4-5fc4-4be6-91b4-03e21a88a636 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:48.846890 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb8feb4-5fc4-4be6-91b4-03e21a88a636 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:48.867051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-ceb8feb4-5fc4-4be6-91b4-03e21a88a636 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:48.867294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c250a03e-49c1-4904-8edc-9280ccc35666 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:48.868796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ceb8feb4-5fc4-4be6-91b4-03e21a88a636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:48.909226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53e842d-73f5-4772-bc5c-702bb85361ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:48.909940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53e842d-73f5-4772-bc5c-702bb85361ee None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:48.910240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53e842d-73f5-4772-bc5c-702bb85361ee None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:48.910509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53e842d-73f5-4772-bc5c-702bb85361ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:48.923651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53e842d-73f5-4772-bc5c-702bb85361ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:48.925329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53e842d-73f5-4772-bc5c-702bb85361ee None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:48.929497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-a53e842d-73f5-4772-bc5c-702bb85361ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:48.943342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:48.944399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:48.945320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:48.945320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:48.945320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:48.949538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 479/968] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:48.950644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53e842d-73f5-4772-bc5c-702bb85361ee None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:48.963814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 490/969] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:48 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:48.997301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:49.064867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53e842d-73f5-4772-bc5c-702bb85361ee None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:49.099494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f95361a-9cc7-405e-be25-899e61d5e964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:49.100110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f95361a-9cc7-405e-be25-899e61d5e964 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:49.100383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f95361a-9cc7-405e-be25-899e61d5e964 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:49.100648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f95361a-9cc7-405e-be25-899e61d5e964 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:49.101288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f95361a-9cc7-405e-be25-899e61d5e964 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:49.101584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f95361a-9cc7-405e-be25-899e61d5e964 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:49.102973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f95361a-9cc7-405e-be25-899e61d5e964 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:49.115391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 480/970] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:49.121075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb8feb4-5fc4-4be6-91b4-03e21a88a636 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:49.224541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb8feb4-5fc4-4be6-91b4-03e21a88a636 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:49.235166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f95361a-9cc7-405e-be25-899e61d5e964 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:49.260605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3c1320-5915-401e-8b5c-1e08da76bb69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:49.261246 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3c1320-5915-401e-8b5c-1e08da76bb69 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:49.261523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3c1320-5915-401e-8b5c-1e08da76bb69 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:49.261790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3c1320-5915-401e-8b5c-1e08da76bb69 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:49.269310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3c1320-5915-401e-8b5c-1e08da76bb69 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:49.270801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3c1320-5915-401e-8b5c-1e08da76bb69 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-0e3c1320-5915-401e-8b5c-1e08da76bb69 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:17:49.282651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:49.283791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:49.284720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:17:49.284720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:17:49.284720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:17:49.284720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:17:49.284720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:17:49.284720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:17:49.284720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:17:49.284720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:49.284720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 491/971] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:49.289872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3c1320-5915-401e-8b5c-1e08da76bb69 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:49.334852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:49.349160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:49.356933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f95361a-9cc7-405e-be25-899e61d5e964 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:49.397262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f3945c-2d56-489b-81e8-2e0948dc77a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:49.397905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f3945c-2d56-489b-81e8-2e0948dc77a3 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:49.398603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f3945c-2d56-489b-81e8-2e0948dc77a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:49.398887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f3945c-2d56-489b-81e8-2e0948dc77a3 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:49.399603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f3945c-2d56-489b-81e8-2e0948dc77a3 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:49.401411 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f3945c-2d56-489b-81e8-2e0948dc77a3 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:49.408895 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3c1320-5915-401e-8b5c-1e08da76bb69 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-77f3945c-2d56-489b-81e8-2e0948dc77a3 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:49.413305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:49.414596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:49.415449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:49.418253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 481/972] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:49 2022] GET /identity/v3/services/object-store => generated 93 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:49.420882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f3945c-2d56-489b-81e8-2e0948dc77a3 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:49.452854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0dcb63-2025-4307-aba8-07f6773525a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:49.453481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0dcb63-2025-4307-aba8-07f6773525a1 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:49.453759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0dcb63-2025-4307-aba8-07f6773525a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:49.454032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0dcb63-2025-4307-aba8-07f6773525a1 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:49.454756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0dcb63-2025-4307-aba8-07f6773525a1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:49.455061 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0dcb63-2025-4307-aba8-07f6773525a1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:49.456505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0dcb63-2025-4307-aba8-07f6773525a1 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:49.471229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 492/973] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:17:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:49.477257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0dcb63-2025-4307-aba8-07f6773525a1 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:49.539015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f3945c-2d56-489b-81e8-2e0948dc77a3 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:49.573995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1a10b9-d1e6-4259-a76e-8462d872556d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:49.574879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1a10b9-d1e6-4259-a76e-8462d872556d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:49.575119 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1a10b9-d1e6-4259-a76e-8462d872556d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:49.575500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1a10b9-d1e6-4259-a76e-8462d872556d None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:49.576145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1a10b9-d1e6-4259-a76e-8462d872556d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:49.576474 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1a10b9-d1e6-4259-a76e-8462d872556d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:49.577878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1a10b9-d1e6-4259-a76e-8462d872556d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:49.587429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 482/974] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Sep 13 15:17:49 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:49.588794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1a10b9-d1e6-4259-a76e-8462d872556d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:49.591507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:49.596903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0dcb63-2025-4307-aba8-07f6773525a1 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:49.620184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:49.640446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4070bfdb-f859-4a0b-96fd-e44ce1624d0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:49.641498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4070bfdb-f859-4a0b-96fd-e44ce1624d0a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:49.641966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4070bfdb-f859-4a0b-96fd-e44ce1624d0a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:49.642481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4070bfdb-f859-4a0b-96fd-e44ce1624d0a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:49.643398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4070bfdb-f859-4a0b-96fd-e44ce1624d0a None admin] RBAC: Adding query filter params (role.id=523ec7cacd954bfcab0ae561f38f09e9, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=eac26eaa3b1a4c8c849a45ed9ba20301) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:49.643911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4070bfdb-f859-4a0b-96fd-e44ce1624d0a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:49.646190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4070bfdb-f859-4a0b-96fd-e44ce1624d0a None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:49.658621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 493/975] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:17:49 2022] GET /identity/v3/role_assignments?user.id=eac26eaa3b1a4c8c849a45ed9ba20301&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=523ec7cacd954bfcab0ae561f38f09e9 => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:49.720310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1a10b9-d1e6-4259-a76e-8462d872556d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:49.759165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07fafc2-a93b-42ef-9478-f75816819e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:49.759712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07fafc2-a93b-42ef-9478-f75816819e4b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:49.759991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07fafc2-a93b-42ef-9478-f75816819e4b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:49.760264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07fafc2-a93b-42ef-9478-f75816819e4b None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:49.760911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07fafc2-a93b-42ef-9478-f75816819e4b None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:49.761214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07fafc2-a93b-42ef-9478-f75816819e4b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:49.762751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07fafc2-a93b-42ef-9478-f75816819e4b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:49.768978 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 483/976] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Sep 13 15:17:49 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:49.774876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4070bfdb-f859-4a0b-96fd-e44ce1624d0a None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:49.890251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4070bfdb-f859-4a0b-96fd-e44ce1624d0a None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:49.924796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b378bf63-ad7f-4f67-8977-7746afece0b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:49.925808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b378bf63-ad7f-4f67-8977-7746afece0b1 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:49.926296 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b378bf63-ad7f-4f67-8977-7746afece0b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:49.926754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b378bf63-ad7f-4f67-8977-7746afece0b1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:49.927639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b378bf63-ad7f-4f67-8977-7746afece0b1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:49.929265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b378bf63-ad7f-4f67-8977-7746afece0b1 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:49.953701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 494/977] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:17:49 2022] POST /identity/v3/endpoints => generated 349 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:17:49.977924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d269c45d-5aca-4470-8919-55ea7b4f70c0 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:49.978266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d269c45d-5aca-4470-8919-55ea7b4f70c0 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:49.978543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d269c45d-5aca-4470-8919-55ea7b4f70c0 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:49.980279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 484/978] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:49.986454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746b7612-880a-482c-9dbc-f4932bb52980 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:49.986982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746b7612-880a-482c-9dbc-f4932bb52980 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:49.987105 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746b7612-880a-482c-9dbc-f4932bb52980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:50.011743 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-746b7612-880a-482c-9dbc-f4932bb52980 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:50.013476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-746b7612-880a-482c-9dbc-f4932bb52980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:50.028926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:50.104248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 495/979] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:49 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:50.378456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d269c45d-5aca-4470-8919-55ea7b4f70c0 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:50.471526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:50.484276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d269c45d-5aca-4470-8919-55ea7b4f70c0 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:50.519877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38396f3-ff3e-4cca-824b-6995f5d91864 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:50.520872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38396f3-ff3e-4cca-824b-6995f5d91864 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:50.521342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38396f3-ff3e-4cca-824b-6995f5d91864 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:50.521824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38396f3-ff3e-4cca-824b-6995f5d91864 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:50.522694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38396f3-ff3e-4cca-824b-6995f5d91864 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:50.524371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38396f3-ff3e-4cca-824b-6995f5d91864 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:50.540479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.sql.core [None req-a38396f3-ff3e-4cca-824b-6995f5d91864 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 13 15:17:50.540479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 13 15:17:50.540479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [parameters: {'user_id': '1862082e5a254857a11fdd3f94b5447a', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 13 15:17:50.540479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81454) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-a38396f3-ff3e-4cca-824b-6995f5d91864 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application result.read() Sep 13 15:17:50.548233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:17:50.549194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application yield resource Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application next(self.gen) Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.session.rollback() Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.force_reraise() Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise self.value Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application session.commit() Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 13 15:17:50.550171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._prepare_impl() Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.session.flush() Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._flush(objects) Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application compat.raise_( Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application flush_context.execute() Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rec.execute(self) Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 13 15:17:50.551081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 13 15:17:50.551989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application result = self._query(query) Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application conn.query(q) Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application result.read() Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 13 15:17:50.554503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [parameters: {'user_id': '1862082e5a254857a11fdd3f94b5447a', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:50.556452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 13 15:17:50.558291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 13 15:17:50.560472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 13 15:17:50.560472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 13 15:17:50.560472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 13 15:17:50.560472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 13 15:17:50.560472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 13 15:17:50.560472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:50.563070 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 485/980] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Sep 13 15:17:50 2022] POST /identity/v3/users => generated 160 bytes in 185 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:50.571507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746b7612-880a-482c-9dbc-f4932bb52980 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:50.679568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746b7612-880a-482c-9dbc-f4932bb52980 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:50.714413 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252f8613-df94-4823-8479-c5128d07d7c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:50.715604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252f8613-df94-4823-8479-c5128d07d7c5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:50.716110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252f8613-df94-4823-8479-c5128d07d7c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:50.716514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252f8613-df94-4823-8479-c5128d07d7c5 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:50.724135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252f8613-df94-4823-8479-c5128d07d7c5 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:50.725734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252f8613-df94-4823-8479-c5128d07d7c5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-252f8613-df94-4823-8479-c5128d07d7c5 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:50.738298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:50.739475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:50.740620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:50.740620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:50.740620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:50.740620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:50.740620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:50.740620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:50.740620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:50.740620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:50.740620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 13 15:17:50.740620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:50.742762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 496/981] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:17:50 2022] GET /identity/v3/users/cinder => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:50.749035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38396f3-ff3e-4cca-824b-6995f5d91864 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:50.853114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38396f3-ff3e-4cca-824b-6995f5d91864 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:50.888238 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347f570a-7f99-4cfd-bf8f-25ae005614ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:50.888861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347f570a-7f99-4cfd-bf8f-25ae005614ab None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:50.889139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347f570a-7f99-4cfd-bf8f-25ae005614ab None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:50.889417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347f570a-7f99-4cfd-bf8f-25ae005614ab None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:50.890143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347f570a-7f99-4cfd-bf8f-25ae005614ab None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:50.890454 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347f570a-7f99-4cfd-bf8f-25ae005614ab None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:50.892115 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347f570a-7f99-4cfd-bf8f-25ae005614ab None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:50.930391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 486/982] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Sep 13 15:17:50 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:51.163963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2739d31-5982-4122-aae1-9a7946cf0b50 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:51.164484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2739d31-5982-4122-aae1-9a7946cf0b50 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:51.164484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2739d31-5982-4122-aae1-9a7946cf0b50 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:51.166002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 497/983] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:51.171673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fefd30-3799-40e5-9cea-8c48e9a5ff90 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:51.172035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fefd30-3799-40e5-9cea-8c48e9a5ff90 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:51.172383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fefd30-3799-40e5-9cea-8c48e9a5ff90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:51.182797 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:51.197556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-14fefd30-3799-40e5-9cea-8c48e9a5ff90 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:51.199132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-14fefd30-3799-40e5-9cea-8c48e9a5ff90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:51.268672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 487/984] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:51 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:51.534782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2739d31-5982-4122-aae1-9a7946cf0b50 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:51.538629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:51.645238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2739d31-5982-4122-aae1-9a7946cf0b50 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:51.684734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad15d94-deb0-41c5-b43b-c6a676c6baca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:51.685369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad15d94-deb0-41c5-b43b-c6a676c6baca None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:51.685661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad15d94-deb0-41c5-b43b-c6a676c6baca None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:51.685898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad15d94-deb0-41c5-b43b-c6a676c6baca None admin] PATH_INFO: `/v3/services/compute` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:51.686540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad15d94-deb0-41c5-b43b-c6a676c6baca None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:51.687914 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad15d94-deb0-41c5-b43b-c6a676c6baca None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-0ad15d94-deb0-41c5-b43b-c6a676c6baca None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:51.697301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:51.698309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:51.699147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 498/985] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:51 2022] GET /identity/v3/services/compute => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:51.706053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fefd30-3799-40e5-9cea-8c48e9a5ff90 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:51.810895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fefd30-3799-40e5-9cea-8c48e9a5ff90 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:51.847707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd226fb-456e-4a8d-a68e-a029c8b32231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:51.848346 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd226fb-456e-4a8d-a68e-a029c8b32231 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:51.848623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd226fb-456e-4a8d-a68e-a029c8b32231 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:51.848877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd226fb-456e-4a8d-a68e-a029c8b32231 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:51.849524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd226fb-456e-4a8d-a68e-a029c8b32231 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:51.849818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd226fb-456e-4a8d-a68e-a029c8b32231 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:51.851264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd226fb-456e-4a8d-a68e-a029c8b32231 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:51.852699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:51.862325 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 488/986] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:51 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:51.864932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad15d94-deb0-41c5-b43b-c6a676c6baca None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:51.936927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:51.961237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07d596d-0109-4896-aa83-7ebd066a231b None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:51.961549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07d596d-0109-4896-aa83-7ebd066a231b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:51.961771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07d596d-0109-4896-aa83-7ebd066a231b None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:51.963173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 489/987] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:51.968464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151c33cf-bd30-47b1-8692-30fda6e28687 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:51.968750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151c33cf-bd30-47b1-8692-30fda6e28687 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:51.969028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151c33cf-bd30-47b1-8692-30fda6e28687 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:51.972868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad15d94-deb0-41c5-b43b-c6a676c6baca None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:51.988651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-151c33cf-bd30-47b1-8692-30fda6e28687 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:51.989852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-151c33cf-bd30-47b1-8692-30fda6e28687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:52.010542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689e1a29-1edd-4c48-9547-5bf458767fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:52.011158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689e1a29-1edd-4c48-9547-5bf458767fb5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:52.011433 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689e1a29-1edd-4c48-9547-5bf458767fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:52.011691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689e1a29-1edd-4c48-9547-5bf458767fb5 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:52.012489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689e1a29-1edd-4c48-9547-5bf458767fb5 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:52.012797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689e1a29-1edd-4c48-9547-5bf458767fb5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:52.014191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689e1a29-1edd-4c48-9547-5bf458767fb5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:52.019429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 499/988] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:51 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:52.024983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689e1a29-1edd-4c48-9547-5bf458767fb5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:52.065503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 490/989] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:51 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:52.125917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689e1a29-1edd-4c48-9547-5bf458767fb5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:52.158781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:52.165199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b47fe8-195c-4750-b102-1e8758fca05b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:52.165853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b47fe8-195c-4750-b102-1e8758fca05b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:52.166146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b47fe8-195c-4750-b102-1e8758fca05b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:52.166390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b47fe8-195c-4750-b102-1e8758fca05b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:52.167097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b47fe8-195c-4750-b102-1e8758fca05b None admin] RBAC: Adding query filter params (interface=public, service_id=a7aa3e74fae749998ca555c59a4ba87c, region_id=RegionOne) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:52.167403 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b47fe8-195c-4750-b102-1e8758fca05b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:52.169120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b47fe8-195c-4750-b102-1e8758fca05b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:52.175045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 500/990] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Sep 13 15:17:52 2022] GET /identity/v3/endpoints?service_id=a7aa3e74fae749998ca555c59a4ba87c&interface=public®ion_id=RegionOne => generated 194 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:52.180658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151c33cf-bd30-47b1-8692-30fda6e28687 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:52.260103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc93bd3-799f-456a-bed5-25022476dacf None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:52.260347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc93bd3-799f-456a-bed5-25022476dacf None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:52.260620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc93bd3-799f-456a-bed5-25022476dacf None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:52.262353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 501/991] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:52.267420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d33df-bdbc-437b-a571-01a3afc6543e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:52.267668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d33df-bdbc-437b-a571-01a3afc6543e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:52.267979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d33df-bdbc-437b-a571-01a3afc6543e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:52.285704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151c33cf-bd30-47b1-8692-30fda6e28687 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:52.286455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-307d33df-bdbc-437b-a571-01a3afc6543e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:52.287635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-307d33df-bdbc-437b-a571-01a3afc6543e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:52.317058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:52.327441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ab068-9f76-4ce3-a947-87011853632c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:52.328069 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ab068-9f76-4ce3-a947-87011853632c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:52.328345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ab068-9f76-4ce3-a947-87011853632c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:52.328615 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ab068-9f76-4ce3-a947-87011853632c None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:52.329223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388ab068-9f76-4ce3-a947-87011853632c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:52.330648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388ab068-9f76-4ce3-a947-87011853632c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:52.346136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 491/992] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Sep 13 15:17:52 2022] GET /identity/v3/services => generated 1727 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:17:52.346136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ab068-9f76-4ce3-a947-87011853632c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:52.369858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 502/993] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:52 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:52.453573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ab068-9f76-4ce3-a947-87011853632c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:52.491745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d9d8fb-8895-4fc6-aff2-3866c592cdf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:52.492399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d9d8fb-8895-4fc6-aff2-3866c592cdf6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:52.492676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d9d8fb-8895-4fc6-aff2-3866c592cdf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:52.492946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d9d8fb-8895-4fc6-aff2-3866c592cdf6 None admin] PATH_INFO: `/v3/services/image` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:52.493534 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d9d8fb-8895-4fc6-aff2-3866c592cdf6 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:52.494962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d9d8fb-8895-4fc6-aff2-3866c592cdf6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-64d9d8fb-8895-4fc6-aff2-3866c592cdf6 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:52.505317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:52.506410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:52.507246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 492/994] 10.222.0.16 () {56 vars in 1105 bytes} [Tue Sep 13 15:17:52 2022] GET /identity/v3/services/image => generated 86 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:52.512773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d33df-bdbc-437b-a571-01a3afc6543e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:52.619999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d33df-bdbc-437b-a571-01a3afc6543e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:52.635363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d9d8fb-8895-4fc6-aff2-3866c592cdf6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:52.653966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:52.665625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f462fc2b-ad9a-48c7-aa3c-1d3743757ce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:52.666512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f462fc2b-ad9a-48c7-aa3c-1d3743757ce3 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:52.666794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f462fc2b-ad9a-48c7-aa3c-1d3743757ce3 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:52.667479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f462fc2b-ad9a-48c7-aa3c-1d3743757ce3 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:52.668392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f462fc2b-ad9a-48c7-aa3c-1d3743757ce3 None admin] RBAC: Adding query filter params (name=image) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:52.668699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f462fc2b-ad9a-48c7-aa3c-1d3743757ce3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:52.670372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f462fc2b-ad9a-48c7-aa3c-1d3743757ce3 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:52.678030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 503/995] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:52 2022] GET /identity/v3/services?name=image => generated 122 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:52.683977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f462fc2b-ad9a-48c7-aa3c-1d3743757ce3 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:52.701265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:52.749946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d9d8fb-8895-4fc6-aff2-3866c592cdf6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:52.786489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083d16a2-bbcf-464e-b9b3-04438b7f7c0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:52.787124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083d16a2-bbcf-464e-b9b3-04438b7f7c0c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:52.787400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083d16a2-bbcf-464e-b9b3-04438b7f7c0c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:52.787671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083d16a2-bbcf-464e-b9b3-04438b7f7c0c None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:52.788256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083d16a2-bbcf-464e-b9b3-04438b7f7c0c None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:52.789638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083d16a2-bbcf-464e-b9b3-04438b7f7c0c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:52.792718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f462fc2b-ad9a-48c7-aa3c-1d3743757ce3 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-083d16a2-bbcf-464e-b9b3-04438b7f7c0c None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:52.799132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:52.800113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:52.801054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 493/996] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:52 2022] GET /identity/v3/services/object-store => generated 93 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:52.806342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083d16a2-bbcf-464e-b9b3-04438b7f7c0c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:52.832479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c6136c-e5c4-4f47-893a-575732a8e940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:52.833099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c6136c-e5c4-4f47-893a-575732a8e940 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:52.833376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c6136c-e5c4-4f47-893a-575732a8e940 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:52.833644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c6136c-e5c4-4f47-893a-575732a8e940 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:52.834315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c6136c-e5c4-4f47-893a-575732a8e940 None admin] RBAC: Adding query filter params (type=image) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:52.834625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c6136c-e5c4-4f47-893a-575732a8e940 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:52.835994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c6136c-e5c4-4f47-893a-575732a8e940 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:52.841165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 504/997] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:52 2022] GET /identity/v3/services?type=image => generated 122 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:52.945010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083d16a2-bbcf-464e-b9b3-04438b7f7c0c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:52.968261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:52.986071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61373a-9b83-473d-9164-bbbe009eb348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:52.986828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61373a-9b83-473d-9164-bbbe009eb348 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:52.987170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61373a-9b83-473d-9164-bbbe009eb348 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:52.987507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61373a-9b83-473d-9164-bbbe009eb348 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:52.988561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61373a-9b83-473d-9164-bbbe009eb348 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:52.988926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61373a-9b83-473d-9164-bbbe009eb348 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:52.990425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61373a-9b83-473d-9164-bbbe009eb348 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:53.001143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 494/998] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Sep 13 15:17:52 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:53.002368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c6136c-e5c4-4f47-893a-575732a8e940 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:53.065771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:53.113224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c6136c-e5c4-4f47-893a-575732a8e940 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:53.149299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ad25d9-563b-4bdf-a0b1-3a3e9eb74dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:53.150073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ad25d9-563b-4bdf-a0b1-3a3e9eb74dc5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:53.150472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ad25d9-563b-4bdf-a0b1-3a3e9eb74dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:53.150838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ad25d9-563b-4bdf-a0b1-3a3e9eb74dc5 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:53.151588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ad25d9-563b-4bdf-a0b1-3a3e9eb74dc5 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:53.151977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ad25d9-563b-4bdf-a0b1-3a3e9eb74dc5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:53.154245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ad25d9-563b-4bdf-a0b1-3a3e9eb74dc5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:53.159510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 505/999] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Sep 13 15:17:53 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:53.165319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61373a-9b83-473d-9164-bbbe009eb348 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:53.273433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61373a-9b83-473d-9164-bbbe009eb348 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:53.308407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7d6374-0d20-4517-8665-6aeae917eb48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:53.309682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d6374-0d20-4517-8665-6aeae917eb48 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:53.309964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d6374-0d20-4517-8665-6aeae917eb48 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:53.310254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d6374-0d20-4517-8665-6aeae917eb48 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:53.310971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7d6374-0d20-4517-8665-6aeae917eb48 None admin] RBAC: Adding query filter params (interface=admin, service_id=21f856fac94a421eb9139fe3a536bb8a, region_id=RegionOne) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:53.311274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7d6374-0d20-4517-8665-6aeae917eb48 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:53.312656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7d6374-0d20-4517-8665-6aeae917eb48 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:53.318025 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 495/1000] 10.222.0.16 () {56 vars in 1249 bytes} [Tue Sep 13 15:17:53 2022] GET /identity/v3/endpoints?service_id=21f856fac94a421eb9139fe3a536bb8a&interface=admin®ion_id=RegionOne => generated 193 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:53.323700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ad25d9-563b-4bdf-a0b1-3a3e9eb74dc5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:53.426849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ad25d9-563b-4bdf-a0b1-3a3e9eb74dc5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:53.444051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:53.463064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afff06a-4c2d-492b-805a-2b067e666ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:53.463677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afff06a-4c2d-492b-805a-2b067e666ac5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:53.463952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afff06a-4c2d-492b-805a-2b067e666ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:53.464191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afff06a-4c2d-492b-805a-2b067e666ac5 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:53.464880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afff06a-4c2d-492b-805a-2b067e666ac5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:53.466788 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afff06a-4c2d-492b-805a-2b067e666ac5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:53.473318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 506/1001] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Sep 13 15:17:53 2022] GET /identity/v3/services => generated 1727 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:17:54.500992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03353e6f-cf18-4a8d-98ba-90f27a078978 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:54.501564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03353e6f-cf18-4a8d-98ba-90f27a078978 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:54.501564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03353e6f-cf18-4a8d-98ba-90f27a078978 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:54.502981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 496/1002] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:54.508300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893634da-f3d0-4625-9cb3-73335bc89134 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:54.508739 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893634da-f3d0-4625-9cb3-73335bc89134 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:54.508819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893634da-f3d0-4625-9cb3-73335bc89134 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:54.527536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-893634da-f3d0-4625-9cb3-73335bc89134 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:54.528714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-893634da-f3d0-4625-9cb3-73335bc89134 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:54.597744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 507/1003] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:54 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:54.860444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03353e6f-cf18-4a8d-98ba-90f27a078978 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:54.918593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:54.975303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03353e6f-cf18-4a8d-98ba-90f27a078978 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:55.008952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778c206-0bab-4fa4-a39b-c6bf9e8e3d33 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:55.009306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778c206-0bab-4fa4-a39b-c6bf9e8e3d33 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:55.009479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778c206-0bab-4fa4-a39b-c6bf9e8e3d33 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:55.011303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 508/1004] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:17:55.012720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa025a0-95dd-4e46-a1bb-c6e79446c6bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:55.013368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa025a0-95dd-4e46-a1bb-c6e79446c6bb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:55.013648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa025a0-95dd-4e46-a1bb-c6e79446c6bb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:55.013920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa025a0-95dd-4e46-a1bb-c6e79446c6bb None admin] PATH_INFO: `/v3/services/compute` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:55.014534 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa025a0-95dd-4e46-a1bb-c6e79446c6bb None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:55.015953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa025a0-95dd-4e46-a1bb-c6e79446c6bb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:55.016125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e135b3-ed30-4e4e-9cc0-b131d8b22452 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:55.016413 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e135b3-ed30-4e4e-9cc0-b131d8b22452 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:55.016689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e135b3-ed30-4e4e-9cc0-b131d8b22452 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-eaa025a0-95dd-4e46-a1bb-c6e79446c6bb None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:55.025242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:55.026498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 13 15:17:55.027407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:55.028713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 497/1005] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:17:54 2022] GET /identity/v3/services/compute => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:55.032116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa025a0-95dd-4e46-a1bb-c6e79446c6bb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:55.035361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-03e135b3-ed30-4e4e-9cc0-b131d8b22452 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:55.036519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-03e135b3-ed30-4e4e-9cc0-b131d8b22452 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:55.108552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 509/1006] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:55 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:55.138388 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa025a0-95dd-4e46-a1bb-c6e79446c6bb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:55.152102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:55.174961 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673b690c-bfa1-408f-bed6-d77afc6cb799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:55.175579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673b690c-bfa1-408f-bed6-d77afc6cb799 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:55.175860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673b690c-bfa1-408f-bed6-d77afc6cb799 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:55.176128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673b690c-bfa1-408f-bed6-d77afc6cb799 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:55.176757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673b690c-bfa1-408f-bed6-d77afc6cb799 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:55.177062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673b690c-bfa1-408f-bed6-d77afc6cb799 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:55.178483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673b690c-bfa1-408f-bed6-d77afc6cb799 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:55.188146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 498/1007] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:55 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:55.189333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e135b3-ed30-4e4e-9cc0-b131d8b22452 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:55.234206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:55.292864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e135b3-ed30-4e4e-9cc0-b131d8b22452 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:55.327835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b127c221-d25d-432f-8cf9-beaeb6e0cd98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:55.328463 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127c221-d25d-432f-8cf9-beaeb6e0cd98 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:55.328724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127c221-d25d-432f-8cf9-beaeb6e0cd98 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:55.328991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127c221-d25d-432f-8cf9-beaeb6e0cd98 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:55.329622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b127c221-d25d-432f-8cf9-beaeb6e0cd98 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:55.329926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b127c221-d25d-432f-8cf9-beaeb6e0cd98 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:55.331364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b127c221-d25d-432f-8cf9-beaeb6e0cd98 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:55.336470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 510/1008] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Sep 13 15:17:55 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:55.342303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673b690c-bfa1-408f-bed6-d77afc6cb799 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:55.374322 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b127c221-d25d-432f-8cf9-beaeb6e0cd98 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:55.450971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673b690c-bfa1-408f-bed6-d77afc6cb799 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:55.484233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b127c221-d25d-432f-8cf9-beaeb6e0cd98 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:55.487958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d533b5c3-850e-45f8-9931-84a11e9bc01f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:55.488579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d533b5c3-850e-45f8-9931-84a11e9bc01f None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:55.488853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d533b5c3-850e-45f8-9931-84a11e9bc01f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:55.489246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d533b5c3-850e-45f8-9931-84a11e9bc01f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:55.489917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d533b5c3-850e-45f8-9931-84a11e9bc01f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:55.491331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d533b5c3-850e-45f8-9931-84a11e9bc01f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:55.492491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:55.515279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 499/1009] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:17:55 2022] POST /identity/v3/endpoints => generated 334 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:17:55.529056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ba43a9-5692-4f5e-8693-0029f7ab4a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:55.529775 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba43a9-5692-4f5e-8693-0029f7ab4a81 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:55.529973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba43a9-5692-4f5e-8693-0029f7ab4a81 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:55.530284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba43a9-5692-4f5e-8693-0029f7ab4a81 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:55.530969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ba43a9-5692-4f5e-8693-0029f7ab4a81 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:55.532381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ba43a9-5692-4f5e-8693-0029f7ab4a81 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:55.554155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 511/1010] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:17:55 2022] POST /identity/v3/services => generated 242 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:17:55.736420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d8058f-5752-4d13-a3a2-263cf33e84f6 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:55.736613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d8058f-5752-4d13-a3a2-263cf33e84f6 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:55.736913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d8058f-5752-4d13-a3a2-263cf33e84f6 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:55.738594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 500/1011] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:55.744175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3509be83-2c55-48ba-91c8-6c6114ab1de4 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:55.744445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3509be83-2c55-48ba-91c8-6c6114ab1de4 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:55.744721 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3509be83-2c55-48ba-91c8-6c6114ab1de4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:55.766117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-3509be83-2c55-48ba-91c8-6c6114ab1de4 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:55.767379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3509be83-2c55-48ba-91c8-6c6114ab1de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:55.847960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 512/1012] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:55 2022] POST /identity/v3/auth/tokens => generated 1822 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:17:56.115689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d8058f-5752-4d13-a3a2-263cf33e84f6 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:56.150258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:56.226348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d8058f-5752-4d13-a3a2-263cf33e84f6 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:56.261321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84b2a02-4951-460b-b806-79d286c19fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:56.262005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84b2a02-4951-460b-b806-79d286c19fef None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:56.262310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84b2a02-4951-460b-b806-79d286c19fef None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:56.262592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84b2a02-4951-460b-b806-79d286c19fef None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:56.263176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84b2a02-4951-460b-b806-79d286c19fef None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:56.264563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84b2a02-4951-460b-b806-79d286c19fef None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-a84b2a02-4951-460b-b806-79d286c19fef None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:56.273833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:56.274842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:56.279177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 501/1013] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Sep 13 15:17:56 2022] GET /identity/v3/services/object-store => generated 93 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:56.282013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3509be83-2c55-48ba-91c8-6c6114ab1de4 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:56.372912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:56.388153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3509be83-2c55-48ba-91c8-6c6114ab1de4 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:56.424725 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce53684-d839-4d38-9c81-e24e80ed208f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:56.425343 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce53684-d839-4d38-9c81-e24e80ed208f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:56.425617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce53684-d839-4d38-9c81-e24e80ed208f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:56.425890 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce53684-d839-4d38-9c81-e24e80ed208f None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:56.426550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce53684-d839-4d38-9c81-e24e80ed208f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:56.426858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce53684-d839-4d38-9c81-e24e80ed208f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:56.428235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce53684-d839-4d38-9c81-e24e80ed208f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:56.433344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 513/1014] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Sep 13 15:17:56 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:56.439071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84b2a02-4951-460b-b806-79d286c19fef None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:56.548978 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:56.556727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84b2a02-4951-460b-b806-79d286c19fef None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:56.592391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bee0dd8-8c21-46e0-96d1-c88332901118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:56.593121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bee0dd8-8c21-46e0-96d1-c88332901118 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:56.593401 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bee0dd8-8c21-46e0-96d1-c88332901118 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:56.593669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bee0dd8-8c21-46e0-96d1-c88332901118 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:56.594488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bee0dd8-8c21-46e0-96d1-c88332901118 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:56.594795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bee0dd8-8c21-46e0-96d1-c88332901118 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:56.596175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bee0dd8-8c21-46e0-96d1-c88332901118 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:56.601153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 502/1015] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Sep 13 15:17:56 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:56.607225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce53684-d839-4d38-9c81-e24e80ed208f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:56.722447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce53684-d839-4d38-9c81-e24e80ed208f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:56.757709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd0c76b-e7b7-4820-9130-533062d755dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:56.758359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0c76b-e7b7-4820-9130-533062d755dd None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:56.758640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0c76b-e7b7-4820-9130-533062d755dd None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:56.758917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0c76b-e7b7-4820-9130-533062d755dd None admin] PATH_INFO: `/v3/endpoints` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:56.759605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd0c76b-e7b7-4820-9130-533062d755dd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:56.760971 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd0c76b-e7b7-4820-9130-533062d755dd None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:56.785686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 514/1016] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:17:56 2022] POST /identity/v3/endpoints => generated 325 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:17:57.822165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e925ef69-9fbe-43c8-8082-3ed394ac505e None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:57.822706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e925ef69-9fbe-43c8-8082-3ed394ac505e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:57.822706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e925ef69-9fbe-43c8-8082-3ed394ac505e None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:57.824089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 503/1017] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:57.829660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437b7cc1-a590-4838-97ac-8b0e4cd429d8 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:57.829800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfb8a2b-cb3b-4369-84b1-547a8eda0fd7 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:57.830196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfb8a2b-cb3b-4369-84b1-547a8eda0fd7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:57.830268 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437b7cc1-a590-4838-97ac-8b0e4cd429d8 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:57.830268 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437b7cc1-a590-4838-97ac-8b0e4cd429d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:57.830420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfb8a2b-cb3b-4369-84b1-547a8eda0fd7 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:57.831502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 515/1018] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:57.836373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab3d5f4-3af0-494d-9146-bc7cf961a667 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:57.836683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab3d5f4-3af0-494d-9146-bc7cf961a667 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:57.836917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab3d5f4-3af0-494d-9146-bc7cf961a667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:57.849683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-437b7cc1-a590-4838-97ac-8b0e4cd429d8 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:57.850901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-437b7cc1-a590-4838-97ac-8b0e4cd429d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:57.855687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-3ab3d5f4-3af0-494d-9146-bc7cf961a667 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:57.856833 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3ab3d5f4-3af0-494d-9146-bc7cf961a667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:57.865150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:57.932338 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 516/1019] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:57 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:57.932903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 504/1020] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:57 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:58.199517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab3d5f4-3af0-494d-9146-bc7cf961a667 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:58.201657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437b7cc1-a590-4838-97ac-8b0e4cd429d8 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:58.289057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:58.306087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437b7cc1-a590-4838-97ac-8b0e4cd429d8 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:58.309301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab3d5f4-3af0-494d-9146-bc7cf961a667 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:58.343489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31817569-3f5d-4f52-a34e-43524ff65203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:58.344128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31817569-3f5d-4f52-a34e-43524ff65203 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:58.344405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31817569-3f5d-4f52-a34e-43524ff65203 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:58.344680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31817569-3f5d-4f52-a34e-43524ff65203 None admin] PATH_INFO: `/v3/services/image` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:58.345190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13868ea7-24e3-4bcd-96c4-5244e7790c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:58.345303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31817569-3f5d-4f52-a34e-43524ff65203 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:58.345834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13868ea7-24e3-4bcd-96c4-5244e7790c40 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:58.346135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13868ea7-24e3-4bcd-96c4-5244e7790c40 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:58.346400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13868ea7-24e3-4bcd-96c4-5244e7790c40 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:58.346666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31817569-3f5d-4f52-a34e-43524ff65203 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:58.353779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-31817569-3f5d-4f52-a34e-43524ff65203 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:58.356849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:58.357886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:58.359668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 505/1021] 10.222.0.16 () {56 vars in 1105 bytes} [Tue Sep 13 15:17:58 2022] GET /identity/v3/services/image => generated 86 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:58.360494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13868ea7-24e3-4bcd-96c4-5244e7790c40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:17:58.364321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31817569-3f5d-4f52-a34e-43524ff65203 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-13868ea7-24e3-4bcd-96c4-5244e7790c40 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:17:58.365545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:58.366826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:17:58.368426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:58.375899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 517/1022] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Sep 13 15:17:58 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:58.377239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13868ea7-24e3-4bcd-96c4-5244e7790c40 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:58.480124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31817569-3f5d-4f52-a34e-43524ff65203 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:58.490607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13868ea7-24e3-4bcd-96c4-5244e7790c40 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:58.515332 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf41c37-9da3-4033-82f4-80a8957dc961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:58.515951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf41c37-9da3-4033-82f4-80a8957dc961 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:58.516228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf41c37-9da3-4033-82f4-80a8957dc961 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:58.516489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf41c37-9da3-4033-82f4-80a8957dc961 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:58.517118 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf41c37-9da3-4033-82f4-80a8957dc961 None admin] RBAC: Adding query filter params (name=image) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:58.517420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf41c37-9da3-4033-82f4-80a8957dc961 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:58.518827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf41c37-9da3-4033-82f4-80a8957dc961 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:58.523790 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 506/1023] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:58 2022] GET /identity/v3/services?name=image => generated 122 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:58.529350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf41c37-9da3-4033-82f4-80a8957dc961 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:58.533938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29541bce-6e46-479a-b5c5-7e252d2797a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:58.534786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29541bce-6e46-479a-b5c5-7e252d2797a1 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:58.535327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29541bce-6e46-479a-b5c5-7e252d2797a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:58.535734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29541bce-6e46-479a-b5c5-7e252d2797a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:58.536542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29541bce-6e46-479a-b5c5-7e252d2797a1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:58.536984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29541bce-6e46-479a-b5c5-7e252d2797a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:58.538586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29541bce-6e46-479a-b5c5-7e252d2797a1 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:58.545926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:58.555107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 518/1024] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Sep 13 15:17:58 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:58.556516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29541bce-6e46-479a-b5c5-7e252d2797a1 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:58.621115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:58.645217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf41c37-9da3-4033-82f4-80a8957dc961 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:58.671085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29541bce-6e46-479a-b5c5-7e252d2797a1 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:58.683601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03e022-e529-4bbe-be65-2f7c87a9d675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:58.684410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03e022-e529-4bbe-be65-2f7c87a9d675 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:58.684690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03e022-e529-4bbe-be65-2f7c87a9d675 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:58.685063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03e022-e529-4bbe-be65-2f7c87a9d675 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:58.685802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d03e022-e529-4bbe-be65-2f7c87a9d675 None admin] RBAC: Adding query filter params (type=image) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:58.686094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d03e022-e529-4bbe-be65-2f7c87a9d675 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:58.687633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d03e022-e529-4bbe-be65-2f7c87a9d675 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:58.697804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 507/1025] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:58 2022] GET /identity/v3/services?type=image => generated 350 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:58.699154 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03e022-e529-4bbe-be65-2f7c87a9d675 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:58.710465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab900b7-e428-4a8e-8523-483daa61d3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:58.711097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab900b7-e428-4a8e-8523-483daa61d3ac None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:58.711373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab900b7-e428-4a8e-8523-483daa61d3ac None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:58.711650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab900b7-e428-4a8e-8523-483daa61d3ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:58.723816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab900b7-e428-4a8e-8523-483daa61d3ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:58.725279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab900b7-e428-4a8e-8523-483daa61d3ac None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-aab900b7-e428-4a8e-8523-483daa61d3ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:58.740458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:58.741474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:58.747672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:58.747672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:58.747672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:58.747672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 519/1026] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:17:58.747672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab900b7-e428-4a8e-8523-483daa61d3ac None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:58.783674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:58.808148 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03e022-e529-4bbe-be65-2f7c87a9d675 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:58.843535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b2484c-5fe0-47fc-a5b0-20003e2f9e4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:58.844214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b2484c-5fe0-47fc-a5b0-20003e2f9e4d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:58.844427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b2484c-5fe0-47fc-a5b0-20003e2f9e4d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:58.844704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b2484c-5fe0-47fc-a5b0-20003e2f9e4d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:58.845468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b2484c-5fe0-47fc-a5b0-20003e2f9e4d None admin] RBAC: Adding query filter params (interface=public, service_id=abc7bb33e5a54380ac4bfca62d19d9b2, region_id=RegionOne) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:58.845785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b2484c-5fe0-47fc-a5b0-20003e2f9e4d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:58.847223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b2484c-5fe0-47fc-a5b0-20003e2f9e4d None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:58.852448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 508/1027] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Sep 13 15:17:58 2022] GET /identity/v3/endpoints?service_id=abc7bb33e5a54380ac4bfca62d19d9b2&interface=public®ion_id=RegionOne => generated 194 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:58.858808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b2484c-5fe0-47fc-a5b0-20003e2f9e4d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:58.864134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab900b7-e428-4a8e-8523-483daa61d3ac None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:58.879647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:58.905071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67638060-9cb3-4792-a4ac-2d7948abed26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:58.906021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67638060-9cb3-4792-a4ac-2d7948abed26 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:58.906332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67638060-9cb3-4792-a4ac-2d7948abed26 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:58.906607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67638060-9cb3-4792-a4ac-2d7948abed26 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:58.907565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67638060-9cb3-4792-a4ac-2d7948abed26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:58.907880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67638060-9cb3-4792-a4ac-2d7948abed26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:58.909257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67638060-9cb3-4792-a4ac-2d7948abed26 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:58.922573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 520/1028] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:58.928216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67638060-9cb3-4792-a4ac-2d7948abed26 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:58.968546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b2484c-5fe0-47fc-a5b0-20003e2f9e4d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:59.008053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eead4015-8db0-4560-90e3-531a59d6916a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:59.008585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eead4015-8db0-4560-90e3-531a59d6916a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:59.008862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eead4015-8db0-4560-90e3-531a59d6916a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:59.009135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eead4015-8db0-4560-90e3-531a59d6916a None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:59.009742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eead4015-8db0-4560-90e3-531a59d6916a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:59.011201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eead4015-8db0-4560-90e3-531a59d6916a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:59.018202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 509/1029] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Sep 13 15:17:58 2022] GET /identity/v3/services => generated 1957 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:17:59.019365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b0fb8d-d585-42a3-b2d1-674b34ad9cc8 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:59.019646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b0fb8d-d585-42a3-b2d1-674b34ad9cc8 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:59.019921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b0fb8d-d585-42a3-b2d1-674b34ad9cc8 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:59.021409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 510/1030] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:17:59 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:17:59.027357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ef77d6-b645-4f58-a0f9-7943b4186a89 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:59.027646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ef77d6-b645-4f58-a0f9-7943b4186a89 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:59.028104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ef77d6-b645-4f58-a0f9-7943b4186a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:59.037364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67638060-9cb3-4792-a4ac-2d7948abed26 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:59.041687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:59.048723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-b8ef77d6-b645-4f58-a0f9-7943b4186a89 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:17:59.050552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b8ef77d6-b645-4f58-a0f9-7943b4186a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:17:59.079099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60323e5b-78ce-4945-8e39-4924ab72e754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:59.079768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60323e5b-78ce-4945-8e39-4924ab72e754 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:59.080046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60323e5b-78ce-4945-8e39-4924ab72e754 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:59.080318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60323e5b-78ce-4945-8e39-4924ab72e754 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:59.088588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60323e5b-78ce-4945-8e39-4924ab72e754 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:59.090907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60323e5b-78ce-4945-8e39-4924ab72e754 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-60323e5b-78ce-4945-8e39-4924ab72e754 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:17:59.103840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:59.104917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:17:59.106339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 521/1031] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Sep 13 15:17:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:59.111776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60323e5b-78ce-4945-8e39-4924ab72e754 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:59.132251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 511/1032] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:17:59 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:17:59.150289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:59.229765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60323e5b-78ce-4945-8e39-4924ab72e754 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:59.265417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cf1fc0-bf6d-4c5a-bce0-cde0b646df8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:59.265941 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf1fc0-bf6d-4c5a-bce0-cde0b646df8d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:59.266243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf1fc0-bf6d-4c5a-bce0-cde0b646df8d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:59.266508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf1fc0-bf6d-4c5a-bce0-cde0b646df8d None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:59.267224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cf1fc0-bf6d-4c5a-bce0-cde0b646df8d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:17:59.267526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cf1fc0-bf6d-4c5a-bce0-cde0b646df8d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:59.268968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cf1fc0-bf6d-4c5a-bce0-cde0b646df8d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:59.296191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 522/1033] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Sep 13 15:17:59 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:17:59.298349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ef77d6-b645-4f58-a0f9-7943b4186a89 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:59.406202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cf1fc0-bf6d-4c5a-bce0-cde0b646df8d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:59.407071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ef77d6-b645-4f58-a0f9-7943b4186a89 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:59.444295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8348edee-f44f-4cfc-a5c4-0a4a7ffe2db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:59.445654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348edee-f44f-4cfc-a5c4-0a4a7ffe2db3 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:59.445654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348edee-f44f-4cfc-a5c4-0a4a7ffe2db3 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:59.445654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348edee-f44f-4cfc-a5c4-0a4a7ffe2db3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:59.449035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:59.515040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:59.520131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8348edee-f44f-4cfc-a5c4-0a4a7ffe2db3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:59.521690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8348edee-f44f-4cfc-a5c4-0a4a7ffe2db3 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:59.526241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cf1fc0-bf6d-4c5a-bce0-cde0b646df8d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:59.568655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed49a304-0997-4693-8578-b46b364e75eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:59.568655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49a304-0997-4693-8578-b46b364e75eb None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:59.568655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49a304-0997-4693-8578-b46b364e75eb None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:59.568655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49a304-0997-4693-8578-b46b364e75eb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:59.569660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed49a304-0997-4693-8578-b46b364e75eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:59.571134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed49a304-0997-4693-8578-b46b364e75eb None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:59.572874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 523/1034] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:17:59.578459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed49a304-0997-4693-8578-b46b364e75eb None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:59.639146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 512/1035] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:17:59 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:17:59.644822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8348edee-f44f-4cfc-a5c4-0a4a7ffe2db3 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:59.680873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:59.703698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed49a304-0997-4693-8578-b46b364e75eb None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:59.746786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe5726f-f2de-4549-8d89-3d36c07e9940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:59.747412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe5726f-f2de-4549-8d89-3d36c07e9940 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:59.747686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe5726f-f2de-4549-8d89-3d36c07e9940 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:59.747957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe5726f-f2de-4549-8d89-3d36c07e9940 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:59.748235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8348edee-f44f-4cfc-a5c4-0a4a7ffe2db3 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:59.748616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe5726f-f2de-4549-8d89-3d36c07e9940 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:59.750334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe5726f-f2de-4549-8d89-3d36c07e9940 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:59.785462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 524/1036] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:17:59 2022] POST /identity/v3/projects => generated 311 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:17:59.787316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5fd5a0-79bc-4c5c-a371-c6f8cbcfdb9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:59.788185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5fd5a0-79bc-4c5c-a371-c6f8cbcfdb9b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:59.788639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5fd5a0-79bc-4c5c-a371-c6f8cbcfdb9b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:59.789081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5fd5a0-79bc-4c5c-a371-c6f8cbcfdb9b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:17:59.802248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5fd5a0-79bc-4c5c-a371-c6f8cbcfdb9b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:17:59.804153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5fd5a0-79bc-4c5c-a371-c6f8cbcfdb9b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-0c5fd5a0-79bc-4c5c-a371-c6f8cbcfdb9b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:17:59.822484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:17:59.823529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:17:59.824780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:17:59.824780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:17:59.824780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:17:59.830530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 513/1037] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:17:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:17:59.831752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe5726f-f2de-4549-8d89-3d36c07e9940 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:17:59.846385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:17:59.953254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe5726f-f2de-4549-8d89-3d36c07e9940 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:17:59.997597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0817bb6f-c78f-4f8b-973e-a0aafe70df98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:17:59.997980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817bb6f-c78f-4f8b-973e-a0aafe70df98 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:17:59.998332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817bb6f-c78f-4f8b-973e-a0aafe70df98 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:17:59.998602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817bb6f-c78f-4f8b-973e-a0aafe70df98 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:00.000157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0817bb6f-c78f-4f8b-973e-a0aafe70df98 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:00.000157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0817bb6f-c78f-4f8b-973e-a0aafe70df98 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:00.001796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0817bb6f-c78f-4f8b-973e-a0aafe70df98 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:00.013466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 525/1038] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:17:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:00.019594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5fd5a0-79bc-4c5c-a371-c6f8cbcfdb9b None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:00.113337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:00.126948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5fd5a0-79bc-4c5c-a371-c6f8cbcfdb9b None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:00.161668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8c1a2f-17fb-4f1e-9c59-7b6ebb9979e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:00.162334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8c1a2f-17fb-4f1e-9c59-7b6ebb9979e5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:00.162605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8c1a2f-17fb-4f1e-9c59-7b6ebb9979e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:00.162923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8c1a2f-17fb-4f1e-9c59-7b6ebb9979e5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:00.169964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8c1a2f-17fb-4f1e-9c59-7b6ebb9979e5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:00.171453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8c1a2f-17fb-4f1e-9c59-7b6ebb9979e5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-3c8c1a2f-17fb-4f1e-9c59-7b6ebb9979e5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:18:00.182559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:00.183577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:00.184479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:18:00.184479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:18:00.184479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:18:00.184479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:18:00.184479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:18:00.184479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:18:00.184479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:18:00.184479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:18:00.184479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 514/1039] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:18:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:00.191320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0817bb6f-c78f-4f8b-973e-a0aafe70df98 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:00.267665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:00.303727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0817bb6f-c78f-4f8b-973e-a0aafe70df98 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:00.339071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb25205-fc91-40c8-bfeb-4f44a301486e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:00.339731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb25205-fc91-40c8-bfeb-4f44a301486e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:00.340093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb25205-fc91-40c8-bfeb-4f44a301486e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:00.340380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb25205-fc91-40c8-bfeb-4f44a301486e None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:00.341081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb25205-fc91-40c8-bfeb-4f44a301486e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:00.341387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb25205-fc91-40c8-bfeb-4f44a301486e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:00.342847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb25205-fc91-40c8-bfeb-4f44a301486e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:00.360199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 526/1040] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:18:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:00.361450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8c1a2f-17fb-4f1e-9c59-7b6ebb9979e5 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:00.461783 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8c1a2f-17fb-4f1e-9c59-7b6ebb9979e5 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:00.497420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0467bba6-75cc-48f8-977f-8f5edf4fb285 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:00.498413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0467bba6-75cc-48f8-977f-8f5edf4fb285 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:00.498885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0467bba6-75cc-48f8-977f-8f5edf4fb285 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:00.499372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0467bba6-75cc-48f8-977f-8f5edf4fb285 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:00.500296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0467bba6-75cc-48f8-977f-8f5edf4fb285 None admin] RBAC: Adding query filter params (role.id=523ec7cacd954bfcab0ae561f38f09e9, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=f027e913f53447a988f74ae4dff84b2c) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:00.500804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0467bba6-75cc-48f8-977f-8f5edf4fb285 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:00.502505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0467bba6-75cc-48f8-977f-8f5edf4fb285 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:00.514420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 515/1041] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:18:00 2022] GET /identity/v3/role_assignments?user.id=f027e913f53447a988f74ae4dff84b2c&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=523ec7cacd954bfcab0ae561f38f09e9 => generated 259 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:18:01.195364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8d4614-c865-4cee-8222-2ba63612bffa None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:01.195920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8d4614-c865-4cee-8222-2ba63612bffa None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:01.195920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8d4614-c865-4cee-8222-2ba63612bffa None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:01.197421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 527/1042] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:01.202942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b5e395-7dac-486a-9a26-89717162568e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:01.203418 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b5e395-7dac-486a-9a26-89717162568e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:01.203504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b5e395-7dac-486a-9a26-89717162568e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:01.222054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-34b5e395-7dac-486a-9a26-89717162568e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:01.223272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-34b5e395-7dac-486a-9a26-89717162568e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:01.231076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:01.294819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 516/1043] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:01 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:01.554994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8d4614-c865-4cee-8222-2ba63612bffa None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:01.667819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8d4614-c865-4cee-8222-2ba63612bffa None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:01.696841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:01.704140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4d9b93-b783-4f82-a17f-d535e5ca4358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:01.704795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4d9b93-b783-4f82-a17f-d535e5ca4358 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:01.705074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4d9b93-b783-4f82-a17f-d535e5ca4358 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:01.705327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4d9b93-b783-4f82-a17f-d535e5ca4358 None admin] PATH_INFO: `/v3/services/image` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:01.705934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4d9b93-b783-4f82-a17f-d535e5ca4358 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:01.707348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4d9b93-b783-4f82-a17f-d535e5ca4358 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-5b4d9b93-b783-4f82-a17f-d535e5ca4358 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:01.716681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:01.717671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 13 15:18:01.718580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:01.722578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 528/1044] 10.222.0.16 () {56 vars in 1105 bytes} [Tue Sep 13 15:18:01 2022] GET /identity/v3/services/image => generated 86 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:01.723861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b5e395-7dac-486a-9a26-89717162568e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:01.826755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b5e395-7dac-486a-9a26-89717162568e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:01.870366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae21826-fb8a-46f9-be04-7f52faa7007f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:01.871061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae21826-fb8a-46f9-be04-7f52faa7007f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:01.871298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae21826-fb8a-46f9-be04-7f52faa7007f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:01.871571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae21826-fb8a-46f9-be04-7f52faa7007f None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:01.872216 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae21826-fb8a-46f9-be04-7f52faa7007f None admin] RBAC: Adding query filter params (name=image) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:01.872517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae21826-fb8a-46f9-be04-7f52faa7007f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:01.873951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae21826-fb8a-46f9-be04-7f52faa7007f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:01.880006 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 517/1045] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:01 2022] GET /identity/v3/services?name=image => generated 122 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:01.886012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4d9b93-b783-4f82-a17f-d535e5ca4358 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:01.977378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98edf516-457c-4d23-ac00-579b9f73fba6 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:01.977667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98edf516-457c-4d23-ac00-579b9f73fba6 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:01.977945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98edf516-457c-4d23-ac00-579b9f73fba6 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:01.979579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 518/1046] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:18:01.984644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e69c523-c6a3-41b3-a655-91402e5fb2ae None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:01.984929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e69c523-c6a3-41b3-a655-91402e5fb2ae None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:01.985200 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e69c523-c6a3-41b3-a655-91402e5fb2ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:01.991120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4d9b93-b783-4f82-a17f-d535e5ca4358 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:01.998811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:02.004683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-6e69c523-c6a3-41b3-a655-91402e5fb2ae None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:02.005913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6e69c523-c6a3-41b3-a655-91402e5fb2ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:02.026307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841ad2a8-4f7e-44d6-88ab-7796be5a1887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:02.027269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841ad2a8-4f7e-44d6-88ab-7796be5a1887 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:02.027545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841ad2a8-4f7e-44d6-88ab-7796be5a1887 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:02.027813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841ad2a8-4f7e-44d6-88ab-7796be5a1887 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:02.028442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841ad2a8-4f7e-44d6-88ab-7796be5a1887 None admin] RBAC: Adding query filter params (type=image) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:02.028747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841ad2a8-4f7e-44d6-88ab-7796be5a1887 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:02.030189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841ad2a8-4f7e-44d6-88ab-7796be5a1887 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:02.035550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 529/1047] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:01 2022] GET /identity/v3/services?type=image => generated 350 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:02.041146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841ad2a8-4f7e-44d6-88ab-7796be5a1887 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:02.077368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 519/1048] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:01 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:02.104868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:02.148337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841ad2a8-4f7e-44d6-88ab-7796be5a1887 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:02.182904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797b08c9-8cea-4d08-83c7-5ea9d35f9228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:02.183575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797b08c9-8cea-4d08-83c7-5ea9d35f9228 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:02.183857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797b08c9-8cea-4d08-83c7-5ea9d35f9228 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:02.184141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797b08c9-8cea-4d08-83c7-5ea9d35f9228 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:02.184801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797b08c9-8cea-4d08-83c7-5ea9d35f9228 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:02.186179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797b08c9-8cea-4d08-83c7-5ea9d35f9228 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:02.214142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 530/1049] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:18:02 2022] POST /identity/v3/endpoints => generated 327 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:18:02.345899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e69c523-c6a3-41b3-a655-91402e5fb2ae None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:02.452563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e69c523-c6a3-41b3-a655-91402e5fb2ae None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:02.491823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:02.500161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e89ed5-555a-47bb-8079-26eb7e3bd055 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:02.500826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e89ed5-555a-47bb-8079-26eb7e3bd055 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:02.501199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e89ed5-555a-47bb-8079-26eb7e3bd055 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:02.501373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e89ed5-555a-47bb-8079-26eb7e3bd055 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:02.502519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e89ed5-555a-47bb-8079-26eb7e3bd055 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:02.504178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e89ed5-555a-47bb-8079-26eb7e3bd055 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:02.506212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 520/1050] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:02.513348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797b08c9-8cea-4d08-83c7-5ea9d35f9228 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:02.630270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797b08c9-8cea-4d08-83c7-5ea9d35f9228 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:02.644281 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76a0a05-06ce-4530-b7c4-ce20947f46f9 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:02.644581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76a0a05-06ce-4530-b7c4-ce20947f46f9 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:02.644822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76a0a05-06ce-4530-b7c4-ce20947f46f9 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:02.646398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 521/1051] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:18:02.647446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:02.651315 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f582ca-e2e6-4927-800c-2bff3017b629 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:02.651464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f582ca-e2e6-4927-800c-2bff3017b629 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:02.651931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f582ca-e2e6-4927-800c-2bff3017b629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:02.669301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bbfc4e-3835-47c1-921b-936b8c991fd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:02.669915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bbfc4e-3835-47c1-921b-936b8c991fd7 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:02.670216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bbfc4e-3835-47c1-921b-936b8c991fd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:02.670485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bbfc4e-3835-47c1-921b-936b8c991fd7 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:02.671132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bbfc4e-3835-47c1-921b-936b8c991fd7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:02.671980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-50f582ca-e2e6-4927-800c-2bff3017b629 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:02.672550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bbfc4e-3835-47c1-921b-936b8c991fd7 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:02.673173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-50f582ca-e2e6-4927-800c-2bff3017b629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:02.706134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 531/1052] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:18:02 2022] POST /identity/v3/users => generated 285 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:18:02.755018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 522/1053] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:03.019454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bbfc4e-3835-47c1-921b-936b8c991fd7 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:03.126866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bbfc4e-3835-47c1-921b-936b8c991fd7 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:03.160854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5255e64-bf22-47e8-a2e4-14e17517e287 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:03.161487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5255e64-bf22-47e8-a2e4-14e17517e287 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:03.161758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5255e64-bf22-47e8-a2e4-14e17517e287 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:03.162010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5255e64-bf22-47e8-a2e4-14e17517e287 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:03.172351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5255e64-bf22-47e8-a2e4-14e17517e287 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-c5255e64-bf22-47e8-a2e4-14e17517e287 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:18:03.177579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:03.178531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:03.179370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 532/1054] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Sep 13 15:18:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:03.186699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f582ca-e2e6-4927-800c-2bff3017b629 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:03.287841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f582ca-e2e6-4927-800c-2bff3017b629 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:03.317536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:03.324500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be0483-62cf-4be8-a74b-cab8e47d3951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:03.325123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be0483-62cf-4be8-a74b-cab8e47d3951 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:03.325396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be0483-62cf-4be8-a74b-cab8e47d3951 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:03.325662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be0483-62cf-4be8-a74b-cab8e47d3951 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:03.326364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27be0483-62cf-4be8-a74b-cab8e47d3951 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:03.326663 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27be0483-62cf-4be8-a74b-cab8e47d3951 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:03.328039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27be0483-62cf-4be8-a74b-cab8e47d3951 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:03.336394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 523/1055] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Sep 13 15:18:03 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:03.342020 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5255e64-bf22-47e8-a2e4-14e17517e287 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:03.376554 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:03.453666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5255e64-bf22-47e8-a2e4-14e17517e287 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:03.488703 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19f1bec-9eb6-4e79-9b5c-1c0765d43f2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:03.489341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19f1bec-9eb6-4e79-9b5c-1c0765d43f2d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:03.489607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19f1bec-9eb6-4e79-9b5c-1c0765d43f2d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:03.489880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19f1bec-9eb6-4e79-9b5c-1c0765d43f2d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:03.500084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19f1bec-9eb6-4e79-9b5c-1c0765d43f2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:03.501561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19f1bec-9eb6-4e79-9b5c-1c0765d43f2d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-d19f1bec-9eb6-4e79-9b5c-1c0765d43f2d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:18:03.516735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:03.517733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:18:03.519235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:18:03.519235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:18:03.519235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:03.519637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 533/1056] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:03.523708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be0483-62cf-4be8-a74b-cab8e47d3951 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:03.635158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be0483-62cf-4be8-a74b-cab8e47d3951 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:03.671628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e470dde8-3687-469c-a504-d7f96937a17b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:03.672251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e470dde8-3687-469c-a504-d7f96937a17b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:03.672529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e470dde8-3687-469c-a504-d7f96937a17b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:03.672800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e470dde8-3687-469c-a504-d7f96937a17b None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:03.673439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e470dde8-3687-469c-a504-d7f96937a17b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:03.673742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e470dde8-3687-469c-a504-d7f96937a17b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:03.675156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e470dde8-3687-469c-a504-d7f96937a17b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:03.687739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 524/1057] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:18:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:18:03.693305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19f1bec-9eb6-4e79-9b5c-1c0765d43f2d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:03.775039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:03.794609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19f1bec-9eb6-4e79-9b5c-1c0765d43f2d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:03.827888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9563d026-e9c6-4b67-a78e-18e7811b02e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:03.828531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9563d026-e9c6-4b67-a78e-18e7811b02e4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:03.828808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9563d026-e9c6-4b67-a78e-18e7811b02e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:03.829078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9563d026-e9c6-4b67-a78e-18e7811b02e4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:03.836626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9563d026-e9c6-4b67-a78e-18e7811b02e4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:03.838063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9563d026-e9c6-4b67-a78e-18e7811b02e4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-9563d026-e9c6-4b67-a78e-18e7811b02e4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:18:03.849734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:03.850950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:03.852281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:03.852281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:03.852281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:18:03.852281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:18:03.852281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:18:03.852281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:18:03.852281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:18:03.852281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:18:03.852281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:18:03.852281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:03.853126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 534/1058] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Sep 13 15:18:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:03.856923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e470dde8-3687-469c-a504-d7f96937a17b None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:03.915544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:03.966090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e470dde8-3687-469c-a504-d7f96937a17b None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:04.001544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cb90d9-0922-4a80-8baf-889ac08c8f88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:04.002151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cb90d9-0922-4a80-8baf-889ac08c8f88 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.002462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cb90d9-0922-4a80-8baf-889ac08c8f88 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.002737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cb90d9-0922-4a80-8baf-889ac08c8f88 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.003454 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cb90d9-0922-4a80-8baf-889ac08c8f88 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:04.003759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cb90d9-0922-4a80-8baf-889ac08c8f88 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:04.005195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cb90d9-0922-4a80-8baf-889ac08c8f88 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:04.028109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 525/1059] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Sep 13 15:18:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:04.029365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9563d026-e9c6-4b67-a78e-18e7811b02e4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:04.135306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9563d026-e9c6-4b67-a78e-18e7811b02e4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:04.170362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4785d35-cdb2-4fb4-a576-ee07c52653ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:04.170914 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4785d35-cdb2-4fb4-a576-ee07c52653ef None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.171184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4785d35-cdb2-4fb4-a576-ee07c52653ef None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.171449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4785d35-cdb2-4fb4-a576-ee07c52653ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.181992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4785d35-cdb2-4fb4-a576-ee07c52653ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:04.183457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4785d35-cdb2-4fb4-a576-ee07c52653ef None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:04.185262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-e4785d35-cdb2-4fb4-a576-ee07c52653ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:18:04.198847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:04.199835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:18:04.200733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:18:04.200733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:18:04.200733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:04.202284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 535/1060] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:18:04.205683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cb90d9-0922-4a80-8baf-889ac08c8f88 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:04.314965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cb90d9-0922-4a80-8baf-889ac08c8f88 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:04.326905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:04.352332 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3219752c-dde0-41cb-a9bb-c14421ed8bb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:04.353174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3219752c-dde0-41cb-a9bb-c14421ed8bb9 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.353596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3219752c-dde0-41cb-a9bb-c14421ed8bb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.354005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3219752c-dde0-41cb-a9bb-c14421ed8bb9 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.354911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3219752c-dde0-41cb-a9bb-c14421ed8bb9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:04.355929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3219752c-dde0-41cb-a9bb-c14421ed8bb9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:04.357455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3219752c-dde0-41cb-a9bb-c14421ed8bb9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:04.370219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 526/1061] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:18:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:04.375634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4785d35-cdb2-4fb4-a576-ee07c52653ef None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:04.391971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f1d0c-fdb3-46ef-a173-890d6da2a23b None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.392267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f1d0c-fdb3-46ef-a173-890d6da2a23b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.392518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f1d0c-fdb3-46ef-a173-890d6da2a23b None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.394005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 527/1062] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:18:04.398665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ab5db-6ef2-45e4-82ce-8f206f7a62be None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.398946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ab5db-6ef2-45e4-82ce-8f206f7a62be None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.399217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ab5db-6ef2-45e4-82ce-8f206f7a62be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.417516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c84ab5db-6ef2-45e4-82ce-8f206f7a62be None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:04.419032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c84ab5db-6ef2-45e4-82ce-8f206f7a62be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:04.484797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4785d35-cdb2-4fb4-a576-ee07c52653ef None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:04.492117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 528/1063] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:18:04.520995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c2943a-cf69-4c52-97f1-2688b8241339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:04.521641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c2943a-cf69-4c52-97f1-2688b8241339 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.521911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c2943a-cf69-4c52-97f1-2688b8241339 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.522199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c2943a-cf69-4c52-97f1-2688b8241339 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.529521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c2943a-cf69-4c52-97f1-2688b8241339 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:04.530991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c2943a-cf69-4c52-97f1-2688b8241339 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-87c2943a-cf69-4c52-97f1-2688b8241339 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:18:04.543519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:04.544412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:04.550580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:18:04.550580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:18:04.550580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:18:04.550580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:18:04.550580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:18:04.550580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:18:04.550580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:18:04.550580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:04.550580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 536/1064] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:18:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:04.551070 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84ab5db-6ef2-45e4-82ce-8f206f7a62be None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:04.664064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:04.667601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84ab5db-6ef2-45e4-82ce-8f206f7a62be None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:04.703966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ca1cb3-182b-4f87-a48b-cbd0212dde5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:04.704579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ca1cb3-182b-4f87-a48b-cbd0212dde5e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.704851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ca1cb3-182b-4f87-a48b-cbd0212dde5e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.705112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ca1cb3-182b-4f87-a48b-cbd0212dde5e None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.705801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ca1cb3-182b-4f87-a48b-cbd0212dde5e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:04.706110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ca1cb3-182b-4f87-a48b-cbd0212dde5e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:04.707569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ca1cb3-182b-4f87-a48b-cbd0212dde5e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:04.720328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 529/1065] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:18:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:04.726240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c2943a-cf69-4c52-97f1-2688b8241339 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:04.755487 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ca1cb3-182b-4f87-a48b-cbd0212dde5e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:04.788590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:04.842243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c2943a-cf69-4c52-97f1-2688b8241339 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:04.861643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ca1cb3-182b-4f87-a48b-cbd0212dde5e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:04.878485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc99b065-a270-4d18-b942-1b04a0ce838e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:04.879152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc99b065-a270-4d18-b942-1b04a0ce838e None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.879430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc99b065-a270-4d18-b942-1b04a0ce838e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.879701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc99b065-a270-4d18-b942-1b04a0ce838e None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/f027e913f53447a988f74ae4dff84b2c/roles/523ec7cacd954bfcab0ae561f38f09e9` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.881813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc99b065-a270-4d18-b942-1b04a0ce838e None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=f027e913f53447a988f74ae4dff84b2c, role_id=523ec7cacd954bfcab0ae561f38f09e9)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:04.883273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc99b065-a270-4d18-b942-1b04a0ce838e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:04.898575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 537/1066] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:18:04 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/f027e913f53447a988f74ae4dff84b2c/roles/523ec7cacd954bfcab0ae561f38f09e9 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:18:04.899432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3eec56-0fce-4050-bb32-1401208ce4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:04.900075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda47ff0-1335-4c2b-9a79-589935431916 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.900183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3eec56-0fce-4050-bb32-1401208ce4b6 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.900339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda47ff0-1335-4c2b-9a79-589935431916 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.900457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3eec56-0fce-4050-bb32-1401208ce4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.900620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda47ff0-1335-4c2b-9a79-589935431916 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.900729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3eec56-0fce-4050-bb32-1401208ce4b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.901909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 538/1067] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:04.907750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e275b124-8690-4df7-ba9c-20b802344fbd None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:04.908051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e275b124-8690-4df7-ba9c-20b802344fbd None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:04.908463 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e275b124-8690-4df7-ba9c-20b802344fbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:04.929328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e275b124-8690-4df7-ba9c-20b802344fbd None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:04.930593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e275b124-8690-4df7-ba9c-20b802344fbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:04.936302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:04.982977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3eec56-0fce-4050-bb32-1401208ce4b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:04.984609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3eec56-0fce-4050-bb32-1401208ce4b6 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:05.010745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 539/1068] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:05.099886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 530/1069] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:18:04 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 13 15:18:05.105847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e275b124-8690-4df7-ba9c-20b802344fbd None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:05.151224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:05.208708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e275b124-8690-4df7-ba9c-20b802344fbd None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:05.242615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006b5def-3502-4827-a808-9b6f772b8456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:05.243351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006b5def-3502-4827-a808-9b6f772b8456 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:05.243690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006b5def-3502-4827-a808-9b6f772b8456 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:05.244026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006b5def-3502-4827-a808-9b6f772b8456 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:05.254422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006b5def-3502-4827-a808-9b6f772b8456 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:05.255951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006b5def-3502-4827-a808-9b6f772b8456 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-006b5def-3502-4827-a808-9b6f772b8456 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:18:05.270806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:05.271932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:18:05.273035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:18:05.273035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:18:05.273035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:05.273760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 540/1070] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:05.279950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3eec56-0fce-4050-bb32-1401208ce4b6 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:05.281834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006b5def-3502-4827-a808-9b6f772b8456 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:05.345155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:05.384249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:05.391093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3eec56-0fce-4050-bb32-1401208ce4b6 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:05.393078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006b5def-3502-4827-a808-9b6f772b8456 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:05.425129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c055e2fb-c5a0-4f7e-91e5-dafdc2242643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:05.425746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c055e2fb-c5a0-4f7e-91e5-dafdc2242643 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:05.426022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c055e2fb-c5a0-4f7e-91e5-dafdc2242643 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:05.426318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c055e2fb-c5a0-4f7e-91e5-dafdc2242643 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:05.427004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c055e2fb-c5a0-4f7e-91e5-dafdc2242643 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:05.427263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c055e2fb-c5a0-4f7e-91e5-dafdc2242643 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:05.428640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c055e2fb-c5a0-4f7e-91e5-dafdc2242643 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:05.428948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d0c1bf-439b-4682-b26e-eb6a401ea91a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:05.429586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d0c1bf-439b-4682-b26e-eb6a401ea91a None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:05.430010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d0c1bf-439b-4682-b26e-eb6a401ea91a None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:05.430371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d0c1bf-439b-4682-b26e-eb6a401ea91a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:05.450132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 531/1071] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:18:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d0c1bf-439b-4682-b26e-eb6a401ea91a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-14d0c1bf-439b-4682-b26e-eb6a401ea91a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:05.450339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:05.451140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:05.452010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 541/1072] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:18:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:05.455074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c055e2fb-c5a0-4f7e-91e5-dafdc2242643 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:05.566526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c055e2fb-c5a0-4f7e-91e5-dafdc2242643 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:05.588260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:05.604409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9205c550-84f9-4707-ad72-b8e4fc2bd751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:05.605314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9205c550-84f9-4707-ad72-b8e4fc2bd751 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:05.605906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9205c550-84f9-4707-ad72-b8e4fc2bd751 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:05.605906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9205c550-84f9-4707-ad72-b8e4fc2bd751 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:05.606605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9205c550-84f9-4707-ad72-b8e4fc2bd751 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:05.606905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9205c550-84f9-4707-ad72-b8e4fc2bd751 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:05.608304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9205c550-84f9-4707-ad72-b8e4fc2bd751 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:05.620958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 532/1073] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:18:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:05.627584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d0c1bf-439b-4682-b26e-eb6a401ea91a None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:05.738482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d0c1bf-439b-4682-b26e-eb6a401ea91a None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:05.776233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8a59fc-bfb8-4e98-abd3-00151b10878c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:05.776953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8a59fc-bfb8-4e98-abd3-00151b10878c None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:05.777232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8a59fc-bfb8-4e98-abd3-00151b10878c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:05.777500 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8a59fc-bfb8-4e98-abd3-00151b10878c None admin] PATH_INFO: `/v3/users/c09899f8a4ae4af78f14b309f10fc675` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:05.791640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8a59fc-bfb8-4e98-abd3-00151b10878c None admin] RBAC: Authorizing `identity:get_user(user_id=c09899f8a4ae4af78f14b309f10fc675)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:05.793191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8a59fc-bfb8-4e98-abd3-00151b10878c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:05.795063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 542/1074] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:05 2022] GET /identity/v3/users/c09899f8a4ae4af78f14b309f10fc675 => generated 285 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:05.801141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9205c550-84f9-4707-ad72-b8e4fc2bd751 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:05.903705 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9205c550-84f9-4707-ad72-b8e4fc2bd751 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:05.939007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abcdabb-12a7-43a8-8f2a-fa49514c47b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:05.939617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abcdabb-12a7-43a8-8f2a-fa49514c47b9 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:05.939893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abcdabb-12a7-43a8-8f2a-fa49514c47b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:05.940161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abcdabb-12a7-43a8-8f2a-fa49514c47b9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:05.990033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:06.011992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abcdabb-12a7-43a8-8f2a-fa49514c47b9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:06.013477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abcdabb-12a7-43a8-8f2a-fa49514c47b9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:06.115952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 533/1075] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:18:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:18:06.122732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8a59fc-bfb8-4e98-abd3-00151b10878c None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:06.139202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:06.226808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8a59fc-bfb8-4e98-abd3-00151b10878c None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:06.260471 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852f0dd8-4dab-4e5b-8d24-3e6a671da85c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:06.261179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852f0dd8-4dab-4e5b-8d24-3e6a671da85c None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:06.261588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852f0dd8-4dab-4e5b-8d24-3e6a671da85c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:06.261861 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852f0dd8-4dab-4e5b-8d24-3e6a671da85c None admin] PATH_INFO: `/v3/projects/2f4b05991f594a69a67a9a6345f4af9c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:06.263010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852f0dd8-4dab-4e5b-8d24-3e6a671da85c None admin] RBAC: Authorizing `identity:get_project(project_id=2f4b05991f594a69a67a9a6345f4af9c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:06.264449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852f0dd8-4dab-4e5b-8d24-3e6a671da85c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:06.266305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 543/1076] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:18:06 2022] GET /identity/v3/projects/2f4b05991f594a69a67a9a6345f4af9c => generated 311 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:06.272341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abcdabb-12a7-43a8-8f2a-fa49514c47b9 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:06.356715 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:06.376007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abcdabb-12a7-43a8-8f2a-fa49514c47b9 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:06.410319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d766ed5-721b-41c8-be97-209e7aa37558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:06.410924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d766ed5-721b-41c8-be97-209e7aa37558 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:06.411197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d766ed5-721b-41c8-be97-209e7aa37558 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:06.411459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d766ed5-721b-41c8-be97-209e7aa37558 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:06.412179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d766ed5-721b-41c8-be97-209e7aa37558 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=2f4b05991f594a69a67a9a6345f4af9c, user.id=c09899f8a4ae4af78f14b309f10fc675) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:06.412478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d766ed5-721b-41c8-be97-209e7aa37558 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:06.413910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d766ed5-721b-41c8-be97-209e7aa37558 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:06.419553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 534/1077] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:18:06 2022] GET /identity/v3/role_assignments?user.id=c09899f8a4ae4af78f14b309f10fc675&scope.project.id=2f4b05991f594a69a67a9a6345f4af9c&role.id=22819414415942d399db534708fe7103 => generated 259 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:07.073727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae002046-ba54-452d-9163-6f408e11c4c2 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:07.074323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae002046-ba54-452d-9163-6f408e11c4c2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:07.074323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae002046-ba54-452d-9163-6f408e11c4c2 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:07.075632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 544/1078] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:07.080997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0870ac1b-a456-4c2f-970c-9d58cfb38491 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:07.081433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0870ac1b-a456-4c2f-970c-9d58cfb38491 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:07.081520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0870ac1b-a456-4c2f-970c-9d58cfb38491 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:07.100096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-0870ac1b-a456-4c2f-970c-9d58cfb38491 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:07.101287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0870ac1b-a456-4c2f-970c-9d58cfb38491 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:07.170286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 535/1079] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:07.434909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae002046-ba54-452d-9163-6f408e11c4c2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:07.518448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:07.538530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae002046-ba54-452d-9163-6f408e11c4c2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:07.572910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07f8b18-4ee8-4fcd-a0cf-c0efac85eccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:07.573560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07f8b18-4ee8-4fcd-a0cf-c0efac85eccf None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:07.573972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07f8b18-4ee8-4fcd-a0cf-c0efac85eccf None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:07.574452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07f8b18-4ee8-4fcd-a0cf-c0efac85eccf None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:07.584707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07f8b18-4ee8-4fcd-a0cf-c0efac85eccf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-b07f8b18-4ee8-4fcd-a0cf-c0efac85eccf None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:18:07.589799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:07.590886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 13 15:18:07.591735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:07.594275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 545/1080] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Sep 13 15:18:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:07.597038 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0870ac1b-a456-4c2f-970c-9d58cfb38491 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:07.677564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:07.700526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0870ac1b-a456-4c2f-970c-9d58cfb38491 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:07.732960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc67d395-43a2-49e4-b54e-e2d63ad111ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:07.733571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc67d395-43a2-49e4-b54e-e2d63ad111ee None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:07.733838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc67d395-43a2-49e4-b54e-e2d63ad111ee None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:07.734129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc67d395-43a2-49e4-b54e-e2d63ad111ee None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:07.734793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc67d395-43a2-49e4-b54e-e2d63ad111ee None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:07.735090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc67d395-43a2-49e4-b54e-e2d63ad111ee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:07.736463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc67d395-43a2-49e4-b54e-e2d63ad111ee None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:07.744475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 536/1081] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Sep 13 15:18:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:07.750512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07f8b18-4ee8-4fcd-a0cf-c0efac85eccf None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:07.849596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07f8b18-4ee8-4fcd-a0cf-c0efac85eccf None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:07.884808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e4f8f-bd8d-455b-a03b-7ca55ec67026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:07.885422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e4f8f-bd8d-455b-a03b-7ca55ec67026 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:07.885695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e4f8f-bd8d-455b-a03b-7ca55ec67026 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:07.886031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e4f8f-bd8d-455b-a03b-7ca55ec67026 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:07.896416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039e4f8f-bd8d-455b-a03b-7ca55ec67026 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:07.897853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039e4f8f-bd8d-455b-a03b-7ca55ec67026 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:07.899589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-039e4f8f-bd8d-455b-a03b-7ca55ec67026 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:18:07.913009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:07.914029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:18:07.914961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:18:07.914961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:18:07.914961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:07.914961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 546/1082] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:07.920106 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc67d395-43a2-49e4-b54e-e2d63ad111ee None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:08.021612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc67d395-43a2-49e4-b54e-e2d63ad111ee None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:08.058948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99597082-bdde-4359-8a44-3f26fdc9770f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:08.059565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99597082-bdde-4359-8a44-3f26fdc9770f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:08.059841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99597082-bdde-4359-8a44-3f26fdc9770f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:08.060111 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99597082-bdde-4359-8a44-3f26fdc9770f None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:08.060756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99597082-bdde-4359-8a44-3f26fdc9770f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:08.061056 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99597082-bdde-4359-8a44-3f26fdc9770f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:08.062460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99597082-bdde-4359-8a44-3f26fdc9770f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:08.069583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:08.074882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 537/1083] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:18:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:08.080758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e4f8f-bd8d-455b-a03b-7ca55ec67026 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:08.189226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e4f8f-bd8d-455b-a03b-7ca55ec67026 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:08.223489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07348d6f-479d-4166-bc4e-97ff922177ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:08.224121 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07348d6f-479d-4166-bc4e-97ff922177ea None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:08.224394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07348d6f-479d-4166-bc4e-97ff922177ea None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:08.224661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07348d6f-479d-4166-bc4e-97ff922177ea None admin] PATH_INFO: `/v3/users/nova` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:08.232078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07348d6f-479d-4166-bc4e-97ff922177ea None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:08.233507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07348d6f-479d-4166-bc4e-97ff922177ea None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-07348d6f-479d-4166-bc4e-97ff922177ea None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:18:08.244711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:08.245762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:08.246741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 547/1084] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Sep 13 15:18:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:18:08.251825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99597082-bdde-4359-8a44-3f26fdc9770f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:08.350714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99597082-bdde-4359-8a44-3f26fdc9770f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:08.385277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd93f65c-e9fd-4c1f-9edb-3b91e83f61c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:08.385916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd93f65c-e9fd-4c1f-9edb-3b91e83f61c5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:08.386190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd93f65c-e9fd-4c1f-9edb-3b91e83f61c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:08.386460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd93f65c-e9fd-4c1f-9edb-3b91e83f61c5 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:08.387207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd93f65c-e9fd-4c1f-9edb-3b91e83f61c5 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:08.387510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd93f65c-e9fd-4c1f-9edb-3b91e83f61c5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:08.388983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd93f65c-e9fd-4c1f-9edb-3b91e83f61c5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:08.408522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 538/1085] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Sep 13 15:18:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:08.414755 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07348d6f-479d-4166-bc4e-97ff922177ea None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:08.473691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:08.520122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4aca3e-1796-4f5e-bf9e-aa8628236ea7 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:08.520421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4aca3e-1796-4f5e-bf9e-aa8628236ea7 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:08.520519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07348d6f-479d-4166-bc4e-97ff922177ea None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:08.520668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4aca3e-1796-4f5e-bf9e-aa8628236ea7 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:08.521989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 539/1086] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:18:08.527223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d114ae32-c572-41a7-af7b-87b1be324288 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:08.527500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d114ae32-c572-41a7-af7b-87b1be324288 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:08.527742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d114ae32-c572-41a7-af7b-87b1be324288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:08.545876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d114ae32-c572-41a7-af7b-87b1be324288 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:08.547637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d114ae32-c572-41a7-af7b-87b1be324288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:08.559265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46571c5d-c11f-48b9-b84e-98348547c1ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:08.559883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46571c5d-c11f-48b9-b84e-98348547c1ae None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:08.560158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46571c5d-c11f-48b9-b84e-98348547c1ae None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:08.560421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46571c5d-c11f-48b9-b84e-98348547c1ae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:08.612636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:08.631821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 540/1087] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:18:08.646260 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46571c5d-c11f-48b9-b84e-98348547c1ae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:08.647717 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46571c5d-c11f-48b9-b84e-98348547c1ae None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:08.701304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:08.747079 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 548/1088] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:18:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:18:08.753110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d114ae32-c572-41a7-af7b-87b1be324288 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:08.854384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d114ae32-c572-41a7-af7b-87b1be324288 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:08.888341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700ce393-7fcb-4114-b919-c815551ccdec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:08.888896 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ce393-7fcb-4114-b919-c815551ccdec None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:08.889173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ce393-7fcb-4114-b919-c815551ccdec None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:08.889440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700ce393-7fcb-4114-b919-c815551ccdec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:08.899677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700ce393-7fcb-4114-b919-c815551ccdec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:08.901130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700ce393-7fcb-4114-b919-c815551ccdec None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:08.904404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46571c5d-c11f-48b9-b84e-98348547c1ae None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-700ce393-7fcb-4114-b919-c815551ccdec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:18:08.918013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:08.919274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:18:08.920485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:18:08.920485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:18:08.920485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:18:08.923905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 541/1089] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:18:08.924691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700ce393-7fcb-4114-b919-c815551ccdec None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:08.966831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:09.005278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46571c5d-c11f-48b9-b84e-98348547c1ae None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:09.029501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700ce393-7fcb-4114-b919-c815551ccdec None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:09.038836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f05ccd3-c148-40fd-ab74-fe92fe5821be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:09.039463 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f05ccd3-c148-40fd-ab74-fe92fe5821be None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:09.039734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f05ccd3-c148-40fd-ab74-fe92fe5821be None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:09.039999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f05ccd3-c148-40fd-ab74-fe92fe5821be None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:09.049974 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f05ccd3-c148-40fd-ab74-fe92fe5821be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-8f05ccd3-c148-40fd-ab74-fe92fe5821be None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:18:09.054677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:09.055634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:09.056479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 549/1090] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:18:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:09.062499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f05ccd3-c148-40fd-ab74-fe92fe5821be None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:09.065400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb3b77-1052-4e15-bc0f-d765f72dbfba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:09.066091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb3b77-1052-4e15-bc0f-d765f72dbfba None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:09.066327 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb3b77-1052-4e15-bc0f-d765f72dbfba None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:09.066599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb3b77-1052-4e15-bc0f-d765f72dbfba None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:09.067249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb3b77-1052-4e15-bc0f-d765f72dbfba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:09.067552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb3b77-1052-4e15-bc0f-d765f72dbfba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:09.068957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb3b77-1052-4e15-bc0f-d765f72dbfba None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:09.079027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:09.084775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 542/1091] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:18:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:18:09.085856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb3b77-1052-4e15-bc0f-d765f72dbfba None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:09.167203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f05ccd3-c148-40fd-ab74-fe92fe5821be None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:09.182589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:09.187007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb3b77-1052-4e15-bc0f-d765f72dbfba None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:09.200931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2136dc36-0da1-425d-af09-33d8b23cd9c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:09.201545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2136dc36-0da1-425d-af09-33d8b23cd9c9 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:09.201820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2136dc36-0da1-425d-af09-33d8b23cd9c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:09.202092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2136dc36-0da1-425d-af09-33d8b23cd9c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:09.202785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2136dc36-0da1-425d-af09-33d8b23cd9c9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:09.203086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2136dc36-0da1-425d-af09-33d8b23cd9c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:09.204470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2136dc36-0da1-425d-af09-33d8b23cd9c9 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:09.212628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 550/1092] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:18:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:18:09.218680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2136dc36-0da1-425d-af09-33d8b23cd9c9 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:09.221101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc086111-da68-4ec7-be01-9fd481e16e58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:09.221739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc086111-da68-4ec7-be01-9fd481e16e58 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:09.222028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc086111-da68-4ec7-be01-9fd481e16e58 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:09.222329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc086111-da68-4ec7-be01-9fd481e16e58 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:09.229079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc086111-da68-4ec7-be01-9fd481e16e58 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:09.230670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc086111-da68-4ec7-be01-9fd481e16e58 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-cc086111-da68-4ec7-be01-9fd481e16e58 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:18:09.242136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:09.243062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:09.244361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:18:09.244361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:18:09.244361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:18:09.244361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:18:09.244361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:18:09.244361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:18:09.244361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:18:09.244361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:18:09.244361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 543/1093] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:18:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:09.249529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc086111-da68-4ec7-be01-9fd481e16e58 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:09.297568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:09.321510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2136dc36-0da1-425d-af09-33d8b23cd9c9 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:09.350729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc086111-da68-4ec7-be01-9fd481e16e58 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:09.356060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411a24e3-25fa-4e05-9507-4bdff0665571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:09.356690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411a24e3-25fa-4e05-9507-4bdff0665571 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:09.356962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411a24e3-25fa-4e05-9507-4bdff0665571 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:09.357225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411a24e3-25fa-4e05-9507-4bdff0665571 None admin] PATH_INFO: `/v3/users/c09899f8a4ae4af78f14b309f10fc675` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:09.358361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411a24e3-25fa-4e05-9507-4bdff0665571 None admin] RBAC: Authorizing `identity:get_user(user_id=c09899f8a4ae4af78f14b309f10fc675)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:09.360070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411a24e3-25fa-4e05-9507-4bdff0665571 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:09.361626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 551/1094] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:09 2022] GET /identity/v3/users/c09899f8a4ae4af78f14b309f10fc675 => generated 285 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:18:09.367405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411a24e3-25fa-4e05-9507-4bdff0665571 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:09.384848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6913a92-bf69-40f3-96a4-a21b404442e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:09.385457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6913a92-bf69-40f3-96a4-a21b404442e0 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:09.385737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6913a92-bf69-40f3-96a4-a21b404442e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:09.386056 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6913a92-bf69-40f3-96a4-a21b404442e0 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:09.386799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6913a92-bf69-40f3-96a4-a21b404442e0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:09.387099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6913a92-bf69-40f3-96a4-a21b404442e0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:09.388532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6913a92-bf69-40f3-96a4-a21b404442e0 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:09.400314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 544/1095] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:18:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:18:09.406353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6913a92-bf69-40f3-96a4-a21b404442e0 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:09.424504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:09.467704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411a24e3-25fa-4e05-9507-4bdff0665571 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:09.501225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a026b412-e0f0-42ab-891a-1dd274664155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:09.501693 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6913a92-bf69-40f3-96a4-a21b404442e0 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:09.501841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a026b412-e0f0-42ab-891a-1dd274664155 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:09.502131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a026b412-e0f0-42ab-891a-1dd274664155 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:09.502424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a026b412-e0f0-42ab-891a-1dd274664155 None admin] PATH_INFO: `/v3/projects/2f4b05991f594a69a67a9a6345f4af9c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:09.504689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a026b412-e0f0-42ab-891a-1dd274664155 None admin] RBAC: Authorizing `identity:get_project(project_id=2f4b05991f594a69a67a9a6345f4af9c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:09.506137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a026b412-e0f0-42ab-891a-1dd274664155 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:09.513089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 552/1096] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:18:09 2022] GET /identity/v3/projects/2f4b05991f594a69a67a9a6345f4af9c => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:18:09.514131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a026b412-e0f0-42ab-891a-1dd274664155 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:09.529325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:09.535823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c65ae7-70a4-4e8a-b823-296cbfcfc3b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:09.536445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c65ae7-70a4-4e8a-b823-296cbfcfc3b9 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:09.536716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c65ae7-70a4-4e8a-b823-296cbfcfc3b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:09.536965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c65ae7-70a4-4e8a-b823-296cbfcfc3b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:09.537682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c65ae7-70a4-4e8a-b823-296cbfcfc3b9 None admin] RBAC: Adding query filter params (role.id=523ec7cacd954bfcab0ae561f38f09e9, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=f027e913f53447a988f74ae4dff84b2c) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:09.537980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c65ae7-70a4-4e8a-b823-296cbfcfc3b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:09.539442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c65ae7-70a4-4e8a-b823-296cbfcfc3b9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:09.545199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 545/1097] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:18:09 2022] GET /identity/v3/role_assignments?user.id=f027e913f53447a988f74ae4dff84b2c&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=523ec7cacd954bfcab0ae561f38f09e9 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:18:09.620682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a026b412-e0f0-42ab-891a-1dd274664155 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:09.662866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049f8091-203a-4b78-a1e4-2c014531f7f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:09.663557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049f8091-203a-4b78-a1e4-2c014531f7f0 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:09.663835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049f8091-203a-4b78-a1e4-2c014531f7f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:09.664111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049f8091-203a-4b78-a1e4-2c014531f7f0 None admin] PATH_INFO: `/v3/projects/2f4b05991f594a69a67a9a6345f4af9c/users/c09899f8a4ae4af78f14b309f10fc675/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:09.666064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049f8091-203a-4b78-a1e4-2c014531f7f0 None admin] RBAC: Authorizing `identity:create_grant(project_id=2f4b05991f594a69a67a9a6345f4af9c, user_id=c09899f8a4ae4af78f14b309f10fc675, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:09.667589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049f8091-203a-4b78-a1e4-2c014531f7f0 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:09.684895 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 553/1098] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:18:09 2022] PUT /identity/v3/projects/2f4b05991f594a69a67a9a6345f4af9c/users/c09899f8a4ae4af78f14b309f10fc675/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:18:11.848201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b90b00-5b1f-4d8e-80f7-b6c9097ef339 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:11.848788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b90b00-5b1f-4d8e-80f7-b6c9097ef339 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:11.848788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b90b00-5b1f-4d8e-80f7-b6c9097ef339 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:11.850077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 546/1099] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:11.855671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8d4d58-66b0-4e32-bf5c-5f7c131a4be5 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:11.856114 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8d4d58-66b0-4e32-bf5c-5f7c131a4be5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:11.856213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8d4d58-66b0-4e32-bf5c-5f7c131a4be5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:11.876799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7a8d4d58-66b0-4e32-bf5c-5f7c131a4be5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:11.877967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7a8d4d58-66b0-4e32-bf5c-5f7c131a4be5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:11.945523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 554/1100] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:12.208585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b90b00-5b1f-4d8e-80f7-b6c9097ef339 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:12.296228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:12.318341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b90b00-5b1f-4d8e-80f7-b6c9097ef339 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:12.356801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a52f6f-02a5-4fa6-ad5e-7f29c92974b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:12.357421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a52f6f-02a5-4fa6-ad5e-7f29c92974b1 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:12.357692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a52f6f-02a5-4fa6-ad5e-7f29c92974b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:12.357959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a52f6f-02a5-4fa6-ad5e-7f29c92974b1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:12.369249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a52f6f-02a5-4fa6-ad5e-7f29c92974b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-b1a52f6f-02a5-4fa6-ad5e-7f29c92974b1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:18:12.373963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:12.375110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:18:12.376439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 547/1101] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:18:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:12.384651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8d4d58-66b0-4e32-bf5c-5f7c131a4be5 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:12.387319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:12.506017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8d4d58-66b0-4e32-bf5c-5f7c131a4be5 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:12.544433 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748cd13a-597a-48ef-a46c-edf92301d856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:12.545033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748cd13a-597a-48ef-a46c-edf92301d856 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:12.545303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748cd13a-597a-48ef-a46c-edf92301d856 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:12.545573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748cd13a-597a-48ef-a46c-edf92301d856 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:12.546257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748cd13a-597a-48ef-a46c-edf92301d856 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:12.546560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748cd13a-597a-48ef-a46c-edf92301d856 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:12.547920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748cd13a-597a-48ef-a46c-edf92301d856 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:12.558798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 555/1102] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:18:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:12.565305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a52f6f-02a5-4fa6-ad5e-7f29c92974b1 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:12.690638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a52f6f-02a5-4fa6-ad5e-7f29c92974b1 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:12.723654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6600326-898e-4f3a-aafa-871dc1e0aa6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:12.724313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6600326-898e-4f3a-aafa-871dc1e0aa6f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:12.724586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6600326-898e-4f3a-aafa-871dc1e0aa6f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:12.724883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6600326-898e-4f3a-aafa-871dc1e0aa6f None admin] PATH_INFO: `/v3/users/c09899f8a4ae4af78f14b309f10fc675` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:12.725964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6600326-898e-4f3a-aafa-871dc1e0aa6f None admin] RBAC: Authorizing `identity:get_user(user_id=c09899f8a4ae4af78f14b309f10fc675)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:12.727639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6600326-898e-4f3a-aafa-871dc1e0aa6f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:12.729216 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 548/1103] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:12 2022] GET /identity/v3/users/c09899f8a4ae4af78f14b309f10fc675 => generated 285 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:12.735384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748cd13a-597a-48ef-a46c-edf92301d856 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:12.802201 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:12.841573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748cd13a-597a-48ef-a46c-edf92301d856 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:12.878279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc12dd53-2bc8-422b-ad7d-c246f68a1c16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:12.878972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc12dd53-2bc8-422b-ad7d-c246f68a1c16 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:12.879152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc12dd53-2bc8-422b-ad7d-c246f68a1c16 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:12.879430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc12dd53-2bc8-422b-ad7d-c246f68a1c16 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:12.951813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc12dd53-2bc8-422b-ad7d-c246f68a1c16 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:12.954010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc12dd53-2bc8-422b-ad7d-c246f68a1c16 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:13.013032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:13.052620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 556/1104] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:18:12 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:18:13.058846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6600326-898e-4f3a-aafa-871dc1e0aa6f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:13.071926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:13.162158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6600326-898e-4f3a-aafa-871dc1e0aa6f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:13.202271 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfde08be-383e-41c6-b2dd-6e88b75471e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:13.202899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfde08be-383e-41c6-b2dd-6e88b75471e0 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:13.203172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfde08be-383e-41c6-b2dd-6e88b75471e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:13.203442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfde08be-383e-41c6-b2dd-6e88b75471e0 None admin] PATH_INFO: `/v3/projects/2f4b05991f594a69a67a9a6345f4af9c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:13.204852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfde08be-383e-41c6-b2dd-6e88b75471e0 None admin] RBAC: Authorizing `identity:get_project(project_id=2f4b05991f594a69a67a9a6345f4af9c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:13.206405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfde08be-383e-41c6-b2dd-6e88b75471e0 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:13.208062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 549/1105] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:18:13 2022] GET /identity/v3/projects/2f4b05991f594a69a67a9a6345f4af9c => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:13.217659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc12dd53-2bc8-422b-ad7d-c246f68a1c16 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:13.325192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc12dd53-2bc8-422b-ad7d-c246f68a1c16 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:13.362583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3a7380-5f18-423a-baab-f29007142a27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:13.363179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3a7380-5f18-423a-baab-f29007142a27 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:13.363452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3a7380-5f18-423a-baab-f29007142a27 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:13.363721 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3a7380-5f18-423a-baab-f29007142a27 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:13.364442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3a7380-5f18-423a-baab-f29007142a27 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=2f4b05991f594a69a67a9a6345f4af9c, user.id=c09899f8a4ae4af78f14b309f10fc675) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:13.364747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3a7380-5f18-423a-baab-f29007142a27 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:13.366209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3a7380-5f18-423a-baab-f29007142a27 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:13.372220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 557/1106] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:18:13 2022] GET /identity/v3/role_assignments?user.id=c09899f8a4ae4af78f14b309f10fc675&scope.project.id=2f4b05991f594a69a67a9a6345f4af9c&role.id=22819414415942d399db534708fe7103 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:15.554004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39dda7f-e9be-49ad-813b-cf411a271c5f None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:15.554914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39dda7f-e9be-49ad-813b-cf411a271c5f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:15.554914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39dda7f-e9be-49ad-813b-cf411a271c5f None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:15.557064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 550/1107] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:15.562910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0124520-14d4-44bf-8ac7-4fbf86d0643a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:15.563326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0124520-14d4-44bf-8ac7-4fbf86d0643a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:15.563448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0124520-14d4-44bf-8ac7-4fbf86d0643a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:15.583561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a0124520-14d4-44bf-8ac7-4fbf86d0643a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:15.584742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a0124520-14d4-44bf-8ac7-4fbf86d0643a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:15.610148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:15.659028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 558/1108] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:15.931517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39dda7f-e9be-49ad-813b-cf411a271c5f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:16.023632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:16.048166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39dda7f-e9be-49ad-813b-cf411a271c5f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:16.083223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0879bb1-f18d-4a78-843d-1c9442eddbaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:16.083843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0879bb1-f18d-4a78-843d-1c9442eddbaa None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:16.084118 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0879bb1-f18d-4a78-843d-1c9442eddbaa None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:16.084387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0879bb1-f18d-4a78-843d-1c9442eddbaa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:16.085538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0879bb1-f18d-4a78-843d-1c9442eddbaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:16.087016 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0879bb1-f18d-4a78-843d-1c9442eddbaa None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:16.088639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 551/1109] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:16.095590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0124520-14d4-44bf-8ac7-4fbf86d0643a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:16.201478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0124520-14d4-44bf-8ac7-4fbf86d0643a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:16.237790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bae05e4-49f5-434a-81df-4957ec0075b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:16.238425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bae05e4-49f5-434a-81df-4957ec0075b6 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:16.238701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bae05e4-49f5-434a-81df-4957ec0075b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:16.238967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bae05e4-49f5-434a-81df-4957ec0075b6 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:16.239619 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bae05e4-49f5-434a-81df-4957ec0075b6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:16.241032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bae05e4-49f5-434a-81df-4957ec0075b6 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:16.266156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 559/1110] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:18:16 2022] POST /identity/v3/users => generated 286 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:18:18.430905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce8f9c2-058e-4049-999a-2f1d2ebc97e0 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:18.431507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce8f9c2-058e-4049-999a-2f1d2ebc97e0 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:18.431507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce8f9c2-058e-4049-999a-2f1d2ebc97e0 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:18.432869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 552/1111] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:18.439440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281b4ebe-b7a9-4bf1-be85-3e3ac5196044 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:18.440146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281b4ebe-b7a9-4bf1-be85-3e3ac5196044 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:18.440324 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281b4ebe-b7a9-4bf1-be85-3e3ac5196044 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:18.468165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-281b4ebe-b7a9-4bf1-be85-3e3ac5196044 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:18.469359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-281b4ebe-b7a9-4bf1-be85-3e3ac5196044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:18.478694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:18.545718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 560/1112] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:18.820094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce8f9c2-058e-4049-999a-2f1d2ebc97e0 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:18.933078 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce8f9c2-058e-4049-999a-2f1d2ebc97e0 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:18.968677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd672f-2282-4e9f-bda3-beac4d24ee38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:18.969296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd672f-2282-4e9f-bda3-beac4d24ee38 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:18.969571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd672f-2282-4e9f-bda3-beac4d24ee38 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:18.969843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafd672f-2282-4e9f-bda3-beac4d24ee38 None admin] PATH_INFO: `/v3/roles/74db8755b1f04d839260ccd3e36deb66` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:18.971369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd672f-2282-4e9f-bda3-beac4d24ee38 None admin] RBAC: Authorizing `identity:get_role(role_id=74db8755b1f04d839260ccd3e36deb66)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:18.972761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fafd672f-2282-4e9f-bda3-beac4d24ee38 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:18.974076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 553/1113] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:18 2022] GET /identity/v3/roles/74db8755b1f04d839260ccd3e36deb66 => generated 223 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:18.980593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281b4ebe-b7a9-4bf1-be85-3e3ac5196044 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:19.091688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281b4ebe-b7a9-4bf1-be85-3e3ac5196044 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:19.144107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f93430-6827-4351-8e33-ae775c9b1fd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:19.144989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f93430-6827-4351-8e33-ae775c9b1fd2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:19.145294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f93430-6827-4351-8e33-ae775c9b1fd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:19.145597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f93430-6827-4351-8e33-ae775c9b1fd2 None admin] PATH_INFO: `/v3/users/f145e61b820c453f8dc0439e2712f2e6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:19.160844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f93430-6827-4351-8e33-ae775c9b1fd2 None admin] RBAC: Authorizing `identity:get_user(user_id=f145e61b820c453f8dc0439e2712f2e6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:19.162307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f93430-6827-4351-8e33-ae775c9b1fd2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:19.163873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 561/1114] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:18 2022] GET /identity/v3/users/f145e61b820c453f8dc0439e2712f2e6 => generated 286 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:19.169804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd672f-2282-4e9f-bda3-beac4d24ee38 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:19.185742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:19.290196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafd672f-2282-4e9f-bda3-beac4d24ee38 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:19.327298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3091ee6-3bff-44e6-8cfa-96a025930347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:19.327893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3091ee6-3bff-44e6-8cfa-96a025930347 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:19.328164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3091ee6-3bff-44e6-8cfa-96a025930347 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:19.328425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3091ee6-3bff-44e6-8cfa-96a025930347 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:19.397713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3091ee6-3bff-44e6-8cfa-96a025930347 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:19.399213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3091ee6-3bff-44e6-8cfa-96a025930347 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:19.414823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:19.500382 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 554/1115] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:18:19 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:18:19.506810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f93430-6827-4351-8e33-ae775c9b1fd2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:19.509892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:19.623375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f93430-6827-4351-8e33-ae775c9b1fd2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:19.659683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab75e5cf-e0d1-4bca-b1bd-fc15f3a237c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:19.660251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab75e5cf-e0d1-4bca-b1bd-fc15f3a237c3 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:19.660527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab75e5cf-e0d1-4bca-b1bd-fc15f3a237c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:19.660795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab75e5cf-e0d1-4bca-b1bd-fc15f3a237c3 None admin] PATH_INFO: `/v3/projects/2f4b05991f594a69a67a9a6345f4af9c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:19.661969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab75e5cf-e0d1-4bca-b1bd-fc15f3a237c3 None admin] RBAC: Authorizing `identity:get_project(project_id=2f4b05991f594a69a67a9a6345f4af9c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:19.663436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab75e5cf-e0d1-4bca-b1bd-fc15f3a237c3 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:19.665076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 562/1116] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:18:19 2022] GET /identity/v3/projects/2f4b05991f594a69a67a9a6345f4af9c => generated 311 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:18:19.672130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3091ee6-3bff-44e6-8cfa-96a025930347 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:19.788636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3091ee6-3bff-44e6-8cfa-96a025930347 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:19.802743 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:19.825802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd21edc1-abe4-40d2-a0cd-f35d1a4757ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:19.826440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd21edc1-abe4-40d2-a0cd-f35d1a4757ce None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:19.826712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd21edc1-abe4-40d2-a0cd-f35d1a4757ce None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:19.826977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd21edc1-abe4-40d2-a0cd-f35d1a4757ce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:19.827695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd21edc1-abe4-40d2-a0cd-f35d1a4757ce None admin] RBAC: Adding query filter params (role.id=74db8755b1f04d839260ccd3e36deb66, scope.project.id=2f4b05991f594a69a67a9a6345f4af9c, user.id=f145e61b820c453f8dc0439e2712f2e6) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:19.827996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd21edc1-abe4-40d2-a0cd-f35d1a4757ce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:19.829428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd21edc1-abe4-40d2-a0cd-f35d1a4757ce None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:19.834905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 555/1117] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:18:19 2022] GET /identity/v3/role_assignments?user.id=f145e61b820c453f8dc0439e2712f2e6&scope.project.id=2f4b05991f594a69a67a9a6345f4af9c&role.id=74db8755b1f04d839260ccd3e36deb66 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:21.970954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164cdba-151b-400d-b511-e6f14e76f90b None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:21.971854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164cdba-151b-400d-b511-e6f14e76f90b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:21.971854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0164cdba-151b-400d-b511-e6f14e76f90b None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:21.974066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 563/1118] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:21.979552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19afb5a1-c702-4e24-beef-0411db07aad3 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:21.979987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19afb5a1-c702-4e24-beef-0411db07aad3 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:21.980088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19afb5a1-c702-4e24-beef-0411db07aad3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:21.999946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-19afb5a1-c702-4e24-beef-0411db07aad3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:22.001128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-19afb5a1-c702-4e24-beef-0411db07aad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:22.070709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 556/1119] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:22.338184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0164cdba-151b-400d-b511-e6f14e76f90b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:22.417199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:22.450347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0164cdba-151b-400d-b511-e6f14e76f90b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:22.492407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bd5abd-09d5-4dd9-b4c3-6168d45e4d7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:22.493028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bd5abd-09d5-4dd9-b4c3-6168d45e4d7d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:22.493309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bd5abd-09d5-4dd9-b4c3-6168d45e4d7d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:22.493579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bd5abd-09d5-4dd9-b4c3-6168d45e4d7d None admin] PATH_INFO: `/v3/roles/74db8755b1f04d839260ccd3e36deb66` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:22.495155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bd5abd-09d5-4dd9-b4c3-6168d45e4d7d None admin] RBAC: Authorizing `identity:get_role(role_id=74db8755b1f04d839260ccd3e36deb66)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:22.496528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2bd5abd-09d5-4dd9-b4c3-6168d45e4d7d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:22.497858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 564/1120] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:22 2022] GET /identity/v3/roles/74db8755b1f04d839260ccd3e36deb66 => generated 223 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:22.503983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19afb5a1-c702-4e24-beef-0411db07aad3 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:22.616341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19afb5a1-c702-4e24-beef-0411db07aad3 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:22.630162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:22.659740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ad6320-583f-42a4-b8c4-1abf215d362b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:22.660384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ad6320-583f-42a4-b8c4-1abf215d362b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:22.660666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ad6320-583f-42a4-b8c4-1abf215d362b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:22.660938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ad6320-583f-42a4-b8c4-1abf215d362b None admin] PATH_INFO: `/v3/users/f145e61b820c453f8dc0439e2712f2e6` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:22.662123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ad6320-583f-42a4-b8c4-1abf215d362b None admin] RBAC: Authorizing `identity:get_user(user_id=f145e61b820c453f8dc0439e2712f2e6)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:22.663594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ad6320-583f-42a4-b8c4-1abf215d362b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:22.665200 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 557/1121] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:22 2022] GET /identity/v3/users/f145e61b820c453f8dc0439e2712f2e6 => generated 286 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:22.671862 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bd5abd-09d5-4dd9-b4c3-6168d45e4d7d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:22.787502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bd5abd-09d5-4dd9-b4c3-6168d45e4d7d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:22.822729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a93ad5-352f-4034-8422-c3f666390307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:22.823236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a93ad5-352f-4034-8422-c3f666390307 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:22.823516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a93ad5-352f-4034-8422-c3f666390307 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:22.823793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a93ad5-352f-4034-8422-c3f666390307 None admin] PATH_INFO: `/v3/projects/2f4b05991f594a69a67a9a6345f4af9c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:22.825052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a93ad5-352f-4034-8422-c3f666390307 None admin] RBAC: Authorizing `identity:get_project(project_id=2f4b05991f594a69a67a9a6345f4af9c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:22.826650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a93ad5-352f-4034-8422-c3f666390307 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:22.828285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 565/1122] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:18:22 2022] GET /identity/v3/projects/2f4b05991f594a69a67a9a6345f4af9c => generated 311 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:22.834641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ad6320-583f-42a4-b8c4-1abf215d362b None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:22.945010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ad6320-583f-42a4-b8c4-1abf215d362b None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:22.979951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b81aeb3-c018-4092-a989-8e838b23cf5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:22.980577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b81aeb3-c018-4092-a989-8e838b23cf5a None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:22.980885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b81aeb3-c018-4092-a989-8e838b23cf5a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:22.981155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b81aeb3-c018-4092-a989-8e838b23cf5a None admin] PATH_INFO: `/v3/projects/2f4b05991f594a69a67a9a6345f4af9c/users/f145e61b820c453f8dc0439e2712f2e6/roles/74db8755b1f04d839260ccd3e36deb66` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:22.983468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b81aeb3-c018-4092-a989-8e838b23cf5a None admin] RBAC: Authorizing `identity:create_grant(project_id=2f4b05991f594a69a67a9a6345f4af9c, user_id=f145e61b820c453f8dc0439e2712f2e6, role_id=74db8755b1f04d839260ccd3e36deb66)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:22.984920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b81aeb3-c018-4092-a989-8e838b23cf5a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:23.001523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 558/1123] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:18:22 2022] PUT /identity/v3/projects/2f4b05991f594a69a67a9a6345f4af9c/users/f145e61b820c453f8dc0439e2712f2e6/roles/74db8755b1f04d839260ccd3e36deb66 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:18:25.198970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de7ef2a-1131-4a39-970f-62ab9bb7bdb6 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:25.199876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de7ef2a-1131-4a39-970f-62ab9bb7bdb6 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:25.199876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de7ef2a-1131-4a39-970f-62ab9bb7bdb6 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:25.201981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 566/1124] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:25.208473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08afdabb-8a38-47e7-b1fe-a13a140a02f0 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:25.209182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08afdabb-8a38-47e7-b1fe-a13a140a02f0 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:25.209334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08afdabb-8a38-47e7-b1fe-a13a140a02f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:25.231516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-08afdabb-8a38-47e7-b1fe-a13a140a02f0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:25.232725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-08afdabb-8a38-47e7-b1fe-a13a140a02f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:25.265052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:25.307864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 559/1125] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:25.575944 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de7ef2a-1131-4a39-970f-62ab9bb7bdb6 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:25.592020 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:25.702896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de7ef2a-1131-4a39-970f-62ab9bb7bdb6 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:25.742909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93eb049c-9f46-424a-a849-427ed992a49d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:25.743535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93eb049c-9f46-424a-a849-427ed992a49d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:25.743806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93eb049c-9f46-424a-a849-427ed992a49d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:25.744074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93eb049c-9f46-424a-a849-427ed992a49d None admin] PATH_INFO: `/v3/roles/74db8755b1f04d839260ccd3e36deb66` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:25.745544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93eb049c-9f46-424a-a849-427ed992a49d None admin] RBAC: Authorizing `identity:get_role(role_id=74db8755b1f04d839260ccd3e36deb66)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:25.746944 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93eb049c-9f46-424a-a849-427ed992a49d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:25.748264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 567/1126] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:25 2022] GET /identity/v3/roles/74db8755b1f04d839260ccd3e36deb66 => generated 223 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:25.754650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08afdabb-8a38-47e7-b1fe-a13a140a02f0 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:25.860414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08afdabb-8a38-47e7-b1fe-a13a140a02f0 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:25.897200 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5643b9a5-ade0-4e86-a9f1-bb289ddfa8c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:25.897823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5643b9a5-ade0-4e86-a9f1-bb289ddfa8c1 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:25.898130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5643b9a5-ade0-4e86-a9f1-bb289ddfa8c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:25.898400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5643b9a5-ade0-4e86-a9f1-bb289ddfa8c1 None admin] PATH_INFO: `/v3/users/f145e61b820c453f8dc0439e2712f2e6` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:25.899555 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5643b9a5-ade0-4e86-a9f1-bb289ddfa8c1 None admin] RBAC: Authorizing `identity:get_user(user_id=f145e61b820c453f8dc0439e2712f2e6)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:25.901086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5643b9a5-ade0-4e86-a9f1-bb289ddfa8c1 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:25.902676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 560/1127] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:25 2022] GET /identity/v3/users/f145e61b820c453f8dc0439e2712f2e6 => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:25.911405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93eb049c-9f46-424a-a849-427ed992a49d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:26.000631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:26.032139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93eb049c-9f46-424a-a849-427ed992a49d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:26.074299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1ba6f6-f93b-4bce-a0a1-481d0879e44d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:26.074905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ba6f6-f93b-4bce-a0a1-481d0879e44d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:26.075179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ba6f6-f93b-4bce-a0a1-481d0879e44d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:26.075447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ba6f6-f93b-4bce-a0a1-481d0879e44d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:26.147674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1ba6f6-f93b-4bce-a0a1-481d0879e44d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:26.149120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1ba6f6-f93b-4bce-a0a1-481d0879e44d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:26.227640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:26.267444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 568/1128] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:18:25 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:18:26.274465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5643b9a5-ade0-4e86-a9f1-bb289ddfa8c1 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:26.316403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:26.398201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5643b9a5-ade0-4e86-a9f1-bb289ddfa8c1 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:26.440720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a846b-11d6-4ac2-a2ae-05c96f5b0b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:26.441371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a846b-11d6-4ac2-a2ae-05c96f5b0b53 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:26.441647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a846b-11d6-4ac2-a2ae-05c96f5b0b53 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:26.441911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a846b-11d6-4ac2-a2ae-05c96f5b0b53 None admin] PATH_INFO: `/v3/projects/2f4b05991f594a69a67a9a6345f4af9c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:26.443116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6a846b-11d6-4ac2-a2ae-05c96f5b0b53 None admin] RBAC: Authorizing `identity:get_project(project_id=2f4b05991f594a69a67a9a6345f4af9c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:26.444565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6a846b-11d6-4ac2-a2ae-05c96f5b0b53 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:26.446244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 561/1129] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:18:26 2022] GET /identity/v3/projects/2f4b05991f594a69a67a9a6345f4af9c => generated 311 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:26.456795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1ba6f6-f93b-4bce-a0a1-481d0879e44d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:26.563297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1ba6f6-f93b-4bce-a0a1-481d0879e44d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:26.597481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb98c85-af6b-47db-adea-88940f5bc9e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:26.598068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb98c85-af6b-47db-adea-88940f5bc9e7 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:26.598373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb98c85-af6b-47db-adea-88940f5bc9e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:26.598635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb98c85-af6b-47db-adea-88940f5bc9e7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:26.599343 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb98c85-af6b-47db-adea-88940f5bc9e7 None admin] RBAC: Adding query filter params (role.id=74db8755b1f04d839260ccd3e36deb66, scope.project.id=2f4b05991f594a69a67a9a6345f4af9c, user.id=f145e61b820c453f8dc0439e2712f2e6) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:26.599649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb98c85-af6b-47db-adea-88940f5bc9e7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:26.601051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb98c85-af6b-47db-adea-88940f5bc9e7 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:26.607044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 569/1130] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:18:26 2022] GET /identity/v3/role_assignments?user.id=f145e61b820c453f8dc0439e2712f2e6&scope.project.id=2f4b05991f594a69a67a9a6345f4af9c&role.id=74db8755b1f04d839260ccd3e36deb66 => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:28.803285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2837fbd-bced-40c9-b544-6663304b30e4 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:28.803883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2837fbd-bced-40c9-b544-6663304b30e4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:28.803883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2837fbd-bced-40c9-b544-6663304b30e4 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:28.806028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 562/1131] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:28 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:28.812754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fd30e9-e50a-4a85-8753-b5e3c4b1eea2 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:28.813192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fd30e9-e50a-4a85-8753-b5e3c4b1eea2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:28.813279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fd30e9-e50a-4a85-8753-b5e3c4b1eea2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:28.841506 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-94fd30e9-e50a-4a85-8753-b5e3c4b1eea2 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:28.842713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-94fd30e9-e50a-4a85-8753-b5e3c4b1eea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:28.877418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:28.915409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 570/1132] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:18:29.184536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2837fbd-bced-40c9-b544-6663304b30e4 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:29.310474 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2837fbd-bced-40c9-b544-6663304b30e4 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:29.312918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:29.350200 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9284e-98b8-44be-b168-dd306c146626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:29.350824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9284e-98b8-44be-b168-dd306c146626 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:29.351104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9284e-98b8-44be-b168-dd306c146626 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:29.351373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9284e-98b8-44be-b168-dd306c146626 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:29.352550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9284e-98b8-44be-b168-dd306c146626 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:29.354156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9284e-98b8-44be-b168-dd306c146626 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:29.356032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 563/1133] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:29.362694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fd30e9-e50a-4a85-8753-b5e3c4b1eea2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:29.464555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fd30e9-e50a-4a85-8753-b5e3c4b1eea2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:29.500023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2425daa0-d5b0-4735-b62d-e14ee041e6f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:29.500623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2425daa0-d5b0-4735-b62d-e14ee041e6f3 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:29.500900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2425daa0-d5b0-4735-b62d-e14ee041e6f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:29.501175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2425daa0-d5b0-4735-b62d-e14ee041e6f3 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:29.501818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2425daa0-d5b0-4735-b62d-e14ee041e6f3 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:29.503283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2425daa0-d5b0-4735-b62d-e14ee041e6f3 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:29.539724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 571/1134] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:18:29 2022] POST /identity/v3/projects => generated 311 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:18:31.768089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb485ef0-4951-4861-829c-b7a5c8e5e60c None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:31.768672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb485ef0-4951-4861-829c-b7a5c8e5e60c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:31.768672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb485ef0-4951-4861-829c-b7a5c8e5e60c None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:31.769995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 564/1135] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:31.776558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43f7cfc-f15a-4c48-83fc-0d5276ee3908 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:31.777296 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43f7cfc-f15a-4c48-83fc-0d5276ee3908 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:31.777465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43f7cfc-f15a-4c48-83fc-0d5276ee3908 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:31.807681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-f43f7cfc-f15a-4c48-83fc-0d5276ee3908 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:31.808894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f43f7cfc-f15a-4c48-83fc-0d5276ee3908 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:31.824165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:31.883556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 572/1136] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:32.155053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb485ef0-4951-4861-829c-b7a5c8e5e60c None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:32.271590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb485ef0-4951-4861-829c-b7a5c8e5e60c None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:32.316217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15de228-706c-485e-a7d6-941974a73863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:32.316850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15de228-706c-485e-a7d6-941974a73863 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:32.317126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15de228-706c-485e-a7d6-941974a73863 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:32.317395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15de228-706c-485e-a7d6-941974a73863 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:32.318562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15de228-706c-485e-a7d6-941974a73863 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:32.320016 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15de228-706c-485e-a7d6-941974a73863 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:32.321558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 565/1137] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:18:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:32.327681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43f7cfc-f15a-4c48-83fc-0d5276ee3908 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:32.428943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43f7cfc-f15a-4c48-83fc-0d5276ee3908 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:32.465324 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5fd253-3a1e-4381-9749-eeb28147c221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:32.465929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5fd253-3a1e-4381-9749-eeb28147c221 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:32.466239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5fd253-3a1e-4381-9749-eeb28147c221 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:32.466504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5fd253-3a1e-4381-9749-eeb28147c221 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:32.467160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5fd253-3a1e-4381-9749-eeb28147c221 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:32.468584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5fd253-3a1e-4381-9749-eeb28147c221 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:32.492492 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 573/1138] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:18:32 2022] POST /identity/v3/users => generated 286 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:18:34.658416 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54328714-61a2-4ea3-ad89-6501ec247fda None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:34.658928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54328714-61a2-4ea3-ad89-6501ec247fda None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:34.658928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54328714-61a2-4ea3-ad89-6501ec247fda None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:34.660331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 566/1139] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:34.666558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec3b9a4-f9e8-4a74-ad98-6a0e6f233ef1 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:34.667002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec3b9a4-f9e8-4a74-ad98-6a0e6f233ef1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:34.667103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec3b9a4-f9e8-4a74-ad98-6a0e6f233ef1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:34.681015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:34.687686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-1ec3b9a4-f9e8-4a74-ad98-6a0e6f233ef1 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:34.689849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1ec3b9a4-f9e8-4a74-ad98-6a0e6f233ef1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:34.761521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 574/1140] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:35.026493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54328714-61a2-4ea3-ad89-6501ec247fda None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:35.087164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:35.163099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54328714-61a2-4ea3-ad89-6501ec247fda None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:35.198441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e91e3c-5f28-468f-817b-f4f524a2cd63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:35.199059 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e91e3c-5f28-468f-817b-f4f524a2cd63 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:35.199339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e91e3c-5f28-468f-817b-f4f524a2cd63 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:35.199612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e91e3c-5f28-468f-817b-f4f524a2cd63 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:35.210305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e91e3c-5f28-468f-817b-f4f524a2cd63 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-88e91e3c-5f28-468f-817b-f4f524a2cd63 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:18:35.215583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:35.216691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:18:35.217919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 567/1141] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:18:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:35.222960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec3b9a4-f9e8-4a74-ad98-6a0e6f233ef1 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:35.336681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec3b9a4-f9e8-4a74-ad98-6a0e6f233ef1 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:35.372245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8198a811-c8b0-4516-aa08-cdac8b7ac101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:35.372848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8198a811-c8b0-4516-aa08-cdac8b7ac101 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:35.373128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8198a811-c8b0-4516-aa08-cdac8b7ac101 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:35.373399 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8198a811-c8b0-4516-aa08-cdac8b7ac101 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:35.374062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8198a811-c8b0-4516-aa08-cdac8b7ac101 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:35.374400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8198a811-c8b0-4516-aa08-cdac8b7ac101 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:35.375775 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8198a811-c8b0-4516-aa08-cdac8b7ac101 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:35.377913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:35.384588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 575/1142] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:18:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:35.390757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e91e3c-5f28-468f-817b-f4f524a2cd63 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:35.490416 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:35.494528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e91e3c-5f28-468f-817b-f4f524a2cd63 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:35.529576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9b3cf4-91fc-4eb8-96ff-6462b473eb3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:35.530216 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9b3cf4-91fc-4eb8-96ff-6462b473eb3f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:35.530496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9b3cf4-91fc-4eb8-96ff-6462b473eb3f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:35.530790 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9b3cf4-91fc-4eb8-96ff-6462b473eb3f None admin] PATH_INFO: `/v3/users/b636c0c852d64597a45c0905077ba886` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:35.547871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9b3cf4-91fc-4eb8-96ff-6462b473eb3f None admin] RBAC: Authorizing `identity:get_user(user_id=b636c0c852d64597a45c0905077ba886)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:35.550043 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9b3cf4-91fc-4eb8-96ff-6462b473eb3f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:35.552509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 568/1143] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:35 2022] GET /identity/v3/users/b636c0c852d64597a45c0905077ba886 => generated 286 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:35.561335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8198a811-c8b0-4516-aa08-cdac8b7ac101 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:35.669528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8198a811-c8b0-4516-aa08-cdac8b7ac101 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:35.705875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e31e320-4f39-4faa-ba37-7e9a20663e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:35.706512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e31e320-4f39-4faa-ba37-7e9a20663e79 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:35.706785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e31e320-4f39-4faa-ba37-7e9a20663e79 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:35.707049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e31e320-4f39-4faa-ba37-7e9a20663e79 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:35.775629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:35.780583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e31e320-4f39-4faa-ba37-7e9a20663e79 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:35.783484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e31e320-4f39-4faa-ba37-7e9a20663e79 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:35.889677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 576/1144] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:18:35 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 13 15:18:35.896215 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9b3cf4-91fc-4eb8-96ff-6462b473eb3f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:36.019460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9b3cf4-91fc-4eb8-96ff-6462b473eb3f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:36.057787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9541a3b-7c29-4703-bd51-b0a7ed85e19b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:36.058441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9541a3b-7c29-4703-bd51-b0a7ed85e19b None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:36.058722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9541a3b-7c29-4703-bd51-b0a7ed85e19b None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:36.058993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9541a3b-7c29-4703-bd51-b0a7ed85e19b None admin] PATH_INFO: `/v3/projects/19e1b5c3f5ea4517922f6f01f338b532` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:36.060173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9541a3b-7c29-4703-bd51-b0a7ed85e19b None admin] RBAC: Authorizing `identity:get_project(project_id=19e1b5c3f5ea4517922f6f01f338b532)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:36.061658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9541a3b-7c29-4703-bd51-b0a7ed85e19b None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:36.063321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 569/1145] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:18:35 2022] GET /identity/v3/projects/19e1b5c3f5ea4517922f6f01f338b532 => generated 311 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:36.073102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e31e320-4f39-4faa-ba37-7e9a20663e79 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:36.179289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:36.184963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e31e320-4f39-4faa-ba37-7e9a20663e79 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:36.220394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af4986d-9e66-44ba-bdb3-f207a62c080f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:36.220988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af4986d-9e66-44ba-bdb3-f207a62c080f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:36.221265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af4986d-9e66-44ba-bdb3-f207a62c080f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:36.221530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af4986d-9e66-44ba-bdb3-f207a62c080f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:36.222269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af4986d-9e66-44ba-bdb3-f207a62c080f None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=19e1b5c3f5ea4517922f6f01f338b532, user.id=b636c0c852d64597a45c0905077ba886) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:36.222579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af4986d-9e66-44ba-bdb3-f207a62c080f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:36.223987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af4986d-9e66-44ba-bdb3-f207a62c080f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:36.229553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 577/1146] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:18:36 2022] GET /identity/v3/role_assignments?user.id=b636c0c852d64597a45c0905077ba886&scope.project.id=19e1b5c3f5ea4517922f6f01f338b532&role.id=22819414415942d399db534708fe7103 => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:38.325205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7948c439-dc2c-41b8-90cd-99977aa7b935 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:38.325888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7948c439-dc2c-41b8-90cd-99977aa7b935 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:38.325888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7948c439-dc2c-41b8-90cd-99977aa7b935 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:38.327116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 570/1147] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:38.333479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b59e6-9add-421d-82a6-9cabc6102c0a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:38.334195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b59e6-9add-421d-82a6-9cabc6102c0a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:38.334310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b59e6-9add-421d-82a6-9cabc6102c0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:38.356340 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-354b59e6-9add-421d-82a6-9cabc6102c0a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:38.357535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-354b59e6-9add-421d-82a6-9cabc6102c0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:38.430042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 578/1148] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:38.707425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7948c439-dc2c-41b8-90cd-99977aa7b935 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:38.736343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:38.814523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7948c439-dc2c-41b8-90cd-99977aa7b935 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:38.849265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282ad02b-697f-4029-8ec1-d021208d8f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:38.849884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282ad02b-697f-4029-8ec1-d021208d8f6f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:38.850166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282ad02b-697f-4029-8ec1-d021208d8f6f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:38.850447 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282ad02b-697f-4029-8ec1-d021208d8f6f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:38.861417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282ad02b-697f-4029-8ec1-d021208d8f6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-282ad02b-697f-4029-8ec1-d021208d8f6f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:18:38.866088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:38.867183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:38.868182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:18:38.870288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 571/1149] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:18:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:38.873449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354b59e6-9add-421d-82a6-9cabc6102c0a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:38.978724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:38.988931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354b59e6-9add-421d-82a6-9cabc6102c0a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:39.023949 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c2696f-4e03-4317-991b-4431c280275b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:39.024551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c2696f-4e03-4317-991b-4431c280275b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:39.024915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c2696f-4e03-4317-991b-4431c280275b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:39.025105 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c2696f-4e03-4317-991b-4431c280275b None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:39.025791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c2696f-4e03-4317-991b-4431c280275b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:39.026089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c2696f-4e03-4317-991b-4431c280275b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:39.027482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c2696f-4e03-4317-991b-4431c280275b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:39.035543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 579/1150] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:18:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:39.042193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282ad02b-697f-4029-8ec1-d021208d8f6f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:39.138419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:39.159858 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282ad02b-697f-4029-8ec1-d021208d8f6f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:39.197015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f912be4-45bc-47ba-9afd-ed630d7ba697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:39.197629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f912be4-45bc-47ba-9afd-ed630d7ba697 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:39.197900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f912be4-45bc-47ba-9afd-ed630d7ba697 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:39.198193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f912be4-45bc-47ba-9afd-ed630d7ba697 None admin] PATH_INFO: `/v3/users/b636c0c852d64597a45c0905077ba886` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:39.199353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f912be4-45bc-47ba-9afd-ed630d7ba697 None admin] RBAC: Authorizing `identity:get_user(user_id=b636c0c852d64597a45c0905077ba886)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:39.200804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f912be4-45bc-47ba-9afd-ed630d7ba697 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:39.202402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 572/1151] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:39 2022] GET /identity/v3/users/b636c0c852d64597a45c0905077ba886 => generated 286 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:39.209175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c2696f-4e03-4317-991b-4431c280275b None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:39.314667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c2696f-4e03-4317-991b-4431c280275b None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:39.351530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2efd465b-2464-430b-be45-c2cdb8277abc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:39.352143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efd465b-2464-430b-be45-c2cdb8277abc None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:39.352407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efd465b-2464-430b-be45-c2cdb8277abc None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:39.352683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efd465b-2464-430b-be45-c2cdb8277abc None admin] PATH_INFO: `/v3/projects/19e1b5c3f5ea4517922f6f01f338b532` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:39.354081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efd465b-2464-430b-be45-c2cdb8277abc None admin] RBAC: Authorizing `identity:get_project(project_id=19e1b5c3f5ea4517922f6f01f338b532)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:39.355541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2efd465b-2464-430b-be45-c2cdb8277abc None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:39.357159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 580/1152] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:18:39 2022] GET /identity/v3/projects/19e1b5c3f5ea4517922f6f01f338b532 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:39.363834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f912be4-45bc-47ba-9afd-ed630d7ba697 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:39.476164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f912be4-45bc-47ba-9afd-ed630d7ba697 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:39.511294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25167bef-cb33-4330-96b0-8bc721618a4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:39.511928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25167bef-cb33-4330-96b0-8bc721618a4f None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:39.512203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25167bef-cb33-4330-96b0-8bc721618a4f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:39.512470 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25167bef-cb33-4330-96b0-8bc721618a4f None admin] PATH_INFO: `/v3/projects/19e1b5c3f5ea4517922f6f01f338b532/users/b636c0c852d64597a45c0905077ba886/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:39.514462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25167bef-cb33-4330-96b0-8bc721618a4f None admin] RBAC: Authorizing `identity:create_grant(project_id=19e1b5c3f5ea4517922f6f01f338b532, user_id=b636c0c852d64597a45c0905077ba886, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:39.515893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25167bef-cb33-4330-96b0-8bc721618a4f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:39.537109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 573/1153] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:18:39 2022] PUT /identity/v3/projects/19e1b5c3f5ea4517922f6f01f338b532/users/b636c0c852d64597a45c0905077ba886/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:18:41.688707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041e8fe5-690c-42af-9691-fd2c447d4e47 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:41.689548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041e8fe5-690c-42af-9691-fd2c447d4e47 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:41.689548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041e8fe5-690c-42af-9691-fd2c447d4e47 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:41.691788 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 581/1154] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:41.699331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa97c3b5-5aac-4cd7-9c89-1e259179b97a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:41.699747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa97c3b5-5aac-4cd7-9c89-1e259179b97a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:41.699878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa97c3b5-5aac-4cd7-9c89-1e259179b97a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:41.725047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-fa97c3b5-5aac-4cd7-9c89-1e259179b97a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:41.726937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fa97c3b5-5aac-4cd7-9c89-1e259179b97a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:41.734941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:41.804792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 574/1155] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:42.091000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041e8fe5-690c-42af-9691-fd2c447d4e47 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:42.129075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:42.210931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041e8fe5-690c-42af-9691-fd2c447d4e47 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:42.246956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fb10d3-b767-438f-ac37-54edf194a2f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:42.247582 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fb10d3-b767-438f-ac37-54edf194a2f2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:42.247862 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fb10d3-b767-438f-ac37-54edf194a2f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:42.248131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fb10d3-b767-438f-ac37-54edf194a2f2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:42.259079 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fb10d3-b767-438f-ac37-54edf194a2f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-c4fb10d3-b767-438f-ac37-54edf194a2f2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:18:42.263881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:42.265018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:42.265929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 582/1156] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:18:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:42.271300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa97c3b5-5aac-4cd7-9c89-1e259179b97a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:42.382189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa97c3b5-5aac-4cd7-9c89-1e259179b97a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:42.417831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6d7671-9422-4c7b-8ad2-20b2c3f44bf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:42.418465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6d7671-9422-4c7b-8ad2-20b2c3f44bf2 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:42.418742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6d7671-9422-4c7b-8ad2-20b2c3f44bf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:42.419006 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6d7671-9422-4c7b-8ad2-20b2c3f44bf2 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:42.419664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6d7671-9422-4c7b-8ad2-20b2c3f44bf2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:42.419961 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6d7671-9422-4c7b-8ad2-20b2c3f44bf2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:42.421343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6d7671-9422-4c7b-8ad2-20b2c3f44bf2 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:42.429593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 575/1157] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:18:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:42.436596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fb10d3-b767-438f-ac37-54edf194a2f2 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:42.528630 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:42.552341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fb10d3-b767-438f-ac37-54edf194a2f2 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:42.590080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f9e6d2-cfe2-45e1-92c9-679a276ca05c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:42.590745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f9e6d2-cfe2-45e1-92c9-679a276ca05c None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:42.591021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f9e6d2-cfe2-45e1-92c9-679a276ca05c None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:42.591287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f9e6d2-cfe2-45e1-92c9-679a276ca05c None admin] PATH_INFO: `/v3/users/b636c0c852d64597a45c0905077ba886` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:42.592487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f9e6d2-cfe2-45e1-92c9-679a276ca05c None admin] RBAC: Authorizing `identity:get_user(user_id=b636c0c852d64597a45c0905077ba886)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:42.593922 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f9e6d2-cfe2-45e1-92c9-679a276ca05c None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:42.595520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 583/1158] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:42 2022] GET /identity/v3/users/b636c0c852d64597a45c0905077ba886 => generated 286 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:42.601707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6d7671-9422-4c7b-8ad2-20b2c3f44bf2 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:42.605918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:42.712238 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6d7671-9422-4c7b-8ad2-20b2c3f44bf2 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:42.752038 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac27a1ca-9b3f-4c56-a770-5f26d6f50b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:42.752645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27a1ca-9b3f-4c56-a770-5f26d6f50b32 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:42.752916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27a1ca-9b3f-4c56-a770-5f26d6f50b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:42.753182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27a1ca-9b3f-4c56-a770-5f26d6f50b32 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:42.839708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac27a1ca-9b3f-4c56-a770-5f26d6f50b32 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:42.841198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac27a1ca-9b3f-4c56-a770-5f26d6f50b32 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:42.843810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:42.948245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 576/1159] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:18:42 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:18:42.955111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f9e6d2-cfe2-45e1-92c9-679a276ca05c None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:43.076572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f9e6d2-cfe2-45e1-92c9-679a276ca05c None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:43.113333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0361fd4-eada-4867-907e-aea63f9bc0ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:43.113955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0361fd4-eada-4867-907e-aea63f9bc0ff None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:43.114252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0361fd4-eada-4867-907e-aea63f9bc0ff None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:43.114522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0361fd4-eada-4867-907e-aea63f9bc0ff None admin] PATH_INFO: `/v3/projects/19e1b5c3f5ea4517922f6f01f338b532` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:43.115757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0361fd4-eada-4867-907e-aea63f9bc0ff None admin] RBAC: Authorizing `identity:get_project(project_id=19e1b5c3f5ea4517922f6f01f338b532)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:43.117191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0361fd4-eada-4867-907e-aea63f9bc0ff None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:43.118935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 584/1160] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:18:42 2022] GET /identity/v3/projects/19e1b5c3f5ea4517922f6f01f338b532 => generated 311 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:43.126658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac27a1ca-9b3f-4c56-a770-5f26d6f50b32 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:43.230824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac27a1ca-9b3f-4c56-a770-5f26d6f50b32 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:43.233048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:43.273306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cde9d87-4a1c-4d63-9caf-b84bc7c521a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:43.273916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cde9d87-4a1c-4d63-9caf-b84bc7c521a3 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:43.274228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cde9d87-4a1c-4d63-9caf-b84bc7c521a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:43.274504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cde9d87-4a1c-4d63-9caf-b84bc7c521a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:43.275261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cde9d87-4a1c-4d63-9caf-b84bc7c521a3 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=19e1b5c3f5ea4517922f6f01f338b532, user.id=b636c0c852d64597a45c0905077ba886) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:43.275565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cde9d87-4a1c-4d63-9caf-b84bc7c521a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:43.277014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cde9d87-4a1c-4d63-9caf-b84bc7c521a3 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:43.283040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 577/1161] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:18:43 2022] GET /identity/v3/role_assignments?user.id=b636c0c852d64597a45c0905077ba886&scope.project.id=19e1b5c3f5ea4517922f6f01f338b532&role.id=22819414415942d399db534708fe7103 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:45.428858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a666ec-ee1b-4046-a487-8ae6e96e1baf None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:45.429469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a666ec-ee1b-4046-a487-8ae6e96e1baf None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:45.429469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a666ec-ee1b-4046-a487-8ae6e96e1baf None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:45.430766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 585/1162] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:45.437252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a47359-31cf-4199-8f0b-2e6e051f11d7 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:45.437941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a47359-31cf-4199-8f0b-2e6e051f11d7 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:45.438166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a47359-31cf-4199-8f0b-2e6e051f11d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:45.460571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-f1a47359-31cf-4199-8f0b-2e6e051f11d7 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:45.461770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f1a47359-31cf-4199-8f0b-2e6e051f11d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:45.533319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 578/1163] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:45.801009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a666ec-ee1b-4046-a487-8ae6e96e1baf None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:45.823969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:45.909218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a666ec-ee1b-4046-a487-8ae6e96e1baf None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:45.944477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0904e4f4-2f67-4c88-a4f8-94a4e04f7eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:45.945107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0904e4f4-2f67-4c88-a4f8-94a4e04f7eb8 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:45.945377 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0904e4f4-2f67-4c88-a4f8-94a4e04f7eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:45.945639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0904e4f4-2f67-4c88-a4f8-94a4e04f7eb8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:46.013624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0904e4f4-2f67-4c88-a4f8-94a4e04f7eb8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:46.015080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0904e4f4-2f67-4c88-a4f8-94a4e04f7eb8 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:46.032639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:46.115740 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 586/1164] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:18:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:18:46.122725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a47359-31cf-4199-8f0b-2e6e051f11d7 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:46.228227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:46.232160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a47359-31cf-4199-8f0b-2e6e051f11d7 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:46.269142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d5b80-56f2-4de8-9093-54db875c3cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:46.269762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d5b80-56f2-4de8-9093-54db875c3cdf None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:46.270035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d5b80-56f2-4de8-9093-54db875c3cdf None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:46.270336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d5b80-56f2-4de8-9093-54db875c3cdf None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:46.277537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583d5b80-56f2-4de8-9093-54db875c3cdf None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:46.279009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583d5b80-56f2-4de8-9093-54db875c3cdf None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-583d5b80-56f2-4de8-9093-54db875c3cdf None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:46.290220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:46.291498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:18:46.292608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 579/1165] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Sep 13 15:18:46 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:46.300049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0904e4f4-2f67-4c88-a4f8-94a4e04f7eb8 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:46.405480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0904e4f4-2f67-4c88-a4f8-94a4e04f7eb8 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:46.422777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:46.440648 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f6a667-18d2-4c11-a43e-2b72eb4f57fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:46.441271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f6a667-18d2-4c11-a43e-2b72eb4f57fc None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:46.441504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f6a667-18d2-4c11-a43e-2b72eb4f57fc None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:46.441772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f6a667-18d2-4c11-a43e-2b72eb4f57fc None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:46.442432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f6a667-18d2-4c11-a43e-2b72eb4f57fc None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:46.442736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f6a667-18d2-4c11-a43e-2b72eb4f57fc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:46.444095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f6a667-18d2-4c11-a43e-2b72eb4f57fc None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:46.451177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 587/1166] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Sep 13 15:18:46 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:48.576422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de306df2-52b5-4578-83c6-f6944e88f9b4 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:48.577317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de306df2-52b5-4578-83c6-f6944e88f9b4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:48.577317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de306df2-52b5-4578-83c6-f6944e88f9b4 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:48.579487 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 580/1167] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:48.587591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be789d6-3184-42cd-85c5-323f9c46a0d5 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:48.588265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be789d6-3184-42cd-85c5-323f9c46a0d5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:48.588265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be789d6-3184-42cd-85c5-323f9c46a0d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:48.608009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0be789d6-3184-42cd-85c5-323f9c46a0d5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:48.609192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0be789d6-3184-42cd-85c5-323f9c46a0d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:48.691548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 588/1168] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 13 15:18:48.963545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de306df2-52b5-4578-83c6-f6944e88f9b4 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:49.074831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de306df2-52b5-4578-83c6-f6944e88f9b4 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:49.111053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbcd3ca-be49-4133-81fd-f6d93fcb40c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:49.111652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbcd3ca-be49-4133-81fd-f6d93fcb40c5 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:49.111927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbcd3ca-be49-4133-81fd-f6d93fcb40c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:49.112193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbcd3ca-be49-4133-81fd-f6d93fcb40c5 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:49.112871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbcd3ca-be49-4133-81fd-f6d93fcb40c5 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:49.114283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbcd3ca-be49-4133-81fd-f6d93fcb40c5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:49.134165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 581/1169] 10.222.0.16 () {60 vars in 1138 bytes} [Tue Sep 13 15:18:48 2022] POST /identity/v3/domains => generated 267 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:18:51.269980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318e46ca-1cb8-4d12-88be-ddb77c4b5e9b None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:51.270606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318e46ca-1cb8-4d12-88be-ddb77c4b5e9b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:51.270606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318e46ca-1cb8-4d12-88be-ddb77c4b5e9b None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:51.271877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 589/1170] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:51.278454 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8a2e19-2f23-423f-b35a-e8565c8cd99d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:51.279189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8a2e19-2f23-423f-b35a-e8565c8cd99d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:51.279377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8a2e19-2f23-423f-b35a-e8565c8cd99d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:51.303523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3d8a2e19-2f23-423f-b35a-e8565c8cd99d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:51.304650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3d8a2e19-2f23-423f-b35a-e8565c8cd99d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:51.320062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:51.385526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 582/1171] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:51.647888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318e46ca-1cb8-4d12-88be-ddb77c4b5e9b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:51.756121 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318e46ca-1cb8-4d12-88be-ddb77c4b5e9b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:51.769316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:51.791466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25692ae5-f5f6-442e-a930-77111a59ff29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:51.792081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25692ae5-f5f6-442e-a930-77111a59ff29 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:51.792352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25692ae5-f5f6-442e-a930-77111a59ff29 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:51.792614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25692ae5-f5f6-442e-a930-77111a59ff29 None admin] PATH_INFO: `/v3/domains/07fb934a29fa4da3bbca178ba54b90da` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:51.803374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25692ae5-f5f6-442e-a930-77111a59ff29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=07fb934a29fa4da3bbca178ba54b90da)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:51.804807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25692ae5-f5f6-442e-a930-77111a59ff29 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:51.806514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 590/1172] 10.222.0.16 () {56 vars in 1183 bytes} [Tue Sep 13 15:18:51 2022] GET /identity/v3/domains/07fb934a29fa4da3bbca178ba54b90da => generated 267 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:51.813232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8a2e19-2f23-423f-b35a-e8565c8cd99d None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:51.937358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8a2e19-2f23-423f-b35a-e8565c8cd99d None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:51.974120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024caa08-ba52-4d1f-9e4c-c9281682f472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:51.974741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024caa08-ba52-4d1f-9e4c-c9281682f472 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:51.975018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024caa08-ba52-4d1f-9e4c-c9281682f472 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:51.975283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024caa08-ba52-4d1f-9e4c-c9281682f472 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:51.975927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024caa08-ba52-4d1f-9e4c-c9281682f472 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:51.977385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024caa08-ba52-4d1f-9e4c-c9281682f472 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:52.002561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:52.009539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 583/1173] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:18:51 2022] POST /identity/v3/projects => generated 361 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:18:54.152886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf76c825-2c44-41f3-9773-f9977ad3a152 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:54.153491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf76c825-2c44-41f3-9773-f9977ad3a152 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:54.153491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf76c825-2c44-41f3-9773-f9977ad3a152 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:54.155515 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 591/1174] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:18:54.161175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dfbf6d-0e7d-430c-9559-f5f54227016e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:54.161613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dfbf6d-0e7d-430c-9559-f5f54227016e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:54.161724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dfbf6d-0e7d-430c-9559-f5f54227016e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:54.181696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-60dfbf6d-0e7d-430c-9559-f5f54227016e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:54.182901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-60dfbf6d-0e7d-430c-9559-f5f54227016e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:54.254479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 584/1175] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:54.524801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf76c825-2c44-41f3-9773-f9977ad3a152 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:54.639827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf76c825-2c44-41f3-9773-f9977ad3a152 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:54.676363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a1d44f-4681-4cee-8c98-8dd1be81969e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:54.676990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a1d44f-4681-4cee-8c98-8dd1be81969e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:54.677265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a1d44f-4681-4cee-8c98-8dd1be81969e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:54.677536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a1d44f-4681-4cee-8c98-8dd1be81969e None admin] PATH_INFO: `/v3/domains/07fb934a29fa4da3bbca178ba54b90da` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:54.678774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a1d44f-4681-4cee-8c98-8dd1be81969e None admin] RBAC: Authorizing `identity:get_domain(domain_id=07fb934a29fa4da3bbca178ba54b90da)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:54.680206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a1d44f-4681-4cee-8c98-8dd1be81969e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:54.681980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 592/1176] 10.222.0.16 () {56 vars in 1183 bytes} [Tue Sep 13 15:18:54 2022] GET /identity/v3/domains/07fb934a29fa4da3bbca178ba54b90da => generated 267 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:54.688317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dfbf6d-0e7d-430c-9559-f5f54227016e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:54.800232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dfbf6d-0e7d-430c-9559-f5f54227016e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:54.832608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:54.840091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f20f8f-7681-47a8-be0c-65581cfd1692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:54.840692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f20f8f-7681-47a8-be0c-65581cfd1692 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:54.840966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f20f8f-7681-47a8-be0c-65581cfd1692 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:54.841243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f20f8f-7681-47a8-be0c-65581cfd1692 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:54.841868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f20f8f-7681-47a8-be0c-65581cfd1692 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:54.843339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f20f8f-7681-47a8-be0c-65581cfd1692 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:54.867740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 585/1177] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:18:54 2022] POST /identity/v3/users => generated 311 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:18:57.092441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a77a3-4153-4927-9c72-4f3367a6473b None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:57.093003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a77a3-4153-4927-9c72-4f3367a6473b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:57.093003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a77a3-4153-4927-9c72-4f3367a6473b None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:57.095081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 593/1178] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:18:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:18:57.104292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e183b6bb-0936-44d9-b74b-94e2e37e1b9a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:57.104701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e183b6bb-0936-44d9-b74b-94e2e37e1b9a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:57.104880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e183b6bb-0936-44d9-b74b-94e2e37e1b9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:57.133872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e183b6bb-0936-44d9-b74b-94e2e37e1b9a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:18:57.133872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e183b6bb-0936-44d9-b74b-94e2e37e1b9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:18:57.210755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 586/1179] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:18:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:18:57.481454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933a77a3-4153-4927-9c72-4f3367a6473b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:57.538284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:57.599855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933a77a3-4153-4927-9c72-4f3367a6473b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:57.634201 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e879964f-1919-4b39-962c-4b94514a7472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:57.634822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e879964f-1919-4b39-962c-4b94514a7472 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:57.635097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e879964f-1919-4b39-962c-4b94514a7472 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:57.635361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e879964f-1919-4b39-962c-4b94514a7472 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:57.645532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e879964f-1919-4b39-962c-4b94514a7472 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-e879964f-1919-4b39-962c-4b94514a7472 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:18:57.650351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:18:57.651470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:18:57.652487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 594/1180] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:18:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:18:57.657632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e183b6bb-0936-44d9-b74b-94e2e37e1b9a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:57.769951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e183b6bb-0936-44d9-b74b-94e2e37e1b9a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:57.788600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:57.811672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42dc34d-195d-4c40-b340-d85f12a4c6db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:57.812318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42dc34d-195d-4c40-b340-d85f12a4c6db None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:57.812595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42dc34d-195d-4c40-b340-d85f12a4c6db None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:57.812861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42dc34d-195d-4c40-b340-d85f12a4c6db None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:57.813529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42dc34d-195d-4c40-b340-d85f12a4c6db None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:57.813831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42dc34d-195d-4c40-b340-d85f12a4c6db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:57.815259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42dc34d-195d-4c40-b340-d85f12a4c6db None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:57.823823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 587/1181] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:18:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:57.830032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e879964f-1919-4b39-962c-4b94514a7472 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:57.938478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:57.941528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e879964f-1919-4b39-962c-4b94514a7472 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:57.974903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac67fbcd-6028-4f4c-a589-29b645a56854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:57.975521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac67fbcd-6028-4f4c-a589-29b645a56854 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:57.975802 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac67fbcd-6028-4f4c-a589-29b645a56854 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:57.976068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac67fbcd-6028-4f4c-a589-29b645a56854 None admin] PATH_INFO: `/v3/users/0e73fa44e2bc407b9d4825ea79bf107b` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:57.989624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac67fbcd-6028-4f4c-a589-29b645a56854 None admin] RBAC: Authorizing `identity:get_user(user_id=0e73fa44e2bc407b9d4825ea79bf107b)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:57.991093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac67fbcd-6028-4f4c-a589-29b645a56854 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:57.992688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 595/1182] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:18:57 2022] GET /identity/v3/users/0e73fa44e2bc407b9d4825ea79bf107b => generated 311 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:57.999849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42dc34d-195d-4c40-b340-d85f12a4c6db None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:58.112961 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42dc34d-195d-4c40-b340-d85f12a4c6db None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:58.167375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a308acfc-2cc4-4f06-82bb-44a0ce363b06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:58.167979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a308acfc-2cc4-4f06-82bb-44a0ce363b06 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:58.168249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a308acfc-2cc4-4f06-82bb-44a0ce363b06 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:58.168497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a308acfc-2cc4-4f06-82bb-44a0ce363b06 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:58.218303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:58.242097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a308acfc-2cc4-4f06-82bb-44a0ce363b06 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:58.243608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a308acfc-2cc4-4f06-82bb-44a0ce363b06 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:58.343446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 588/1183] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:18:57 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:18:58.349736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac67fbcd-6028-4f4c-a589-29b645a56854 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:58.456028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac67fbcd-6028-4f4c-a589-29b645a56854 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:58.493883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c454c04-bbd5-4507-8a41-1a38e41f1f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:58.494254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c454c04-bbd5-4507-8a41-1a38e41f1f55 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:58.494551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c454c04-bbd5-4507-8a41-1a38e41f1f55 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:58.494822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c454c04-bbd5-4507-8a41-1a38e41f1f55 None admin] PATH_INFO: `/v3/projects/99d46889c847422eafee000f4388dbed` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:58.496002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c454c04-bbd5-4507-8a41-1a38e41f1f55 None admin] RBAC: Authorizing `identity:get_project(project_id=99d46889c847422eafee000f4388dbed)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:58.497435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c454c04-bbd5-4507-8a41-1a38e41f1f55 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:58.499095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 596/1184] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:18:58 2022] GET /identity/v3/projects/99d46889c847422eafee000f4388dbed => generated 361 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:18:58.505649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a308acfc-2cc4-4f06-82bb-44a0ce363b06 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:18:58.596605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:18:58.621591 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a308acfc-2cc4-4f06-82bb-44a0ce363b06 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:18:58.658483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4760fe3-89d5-4999-bb5d-0b2f0f4fc1ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:18:58.659101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4760fe3-89d5-4999-bb5d-0b2f0f4fc1ba None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:18:58.659379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4760fe3-89d5-4999-bb5d-0b2f0f4fc1ba None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:18:58.659648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4760fe3-89d5-4999-bb5d-0b2f0f4fc1ba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:18:58.660403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4760fe3-89d5-4999-bb5d-0b2f0f4fc1ba None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=99d46889c847422eafee000f4388dbed, user.id=0e73fa44e2bc407b9d4825ea79bf107b) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:18:58.660709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4760fe3-89d5-4999-bb5d-0b2f0f4fc1ba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:18:58.662175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4760fe3-89d5-4999-bb5d-0b2f0f4fc1ba None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:18:58.668316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 589/1185] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:18:58 2022] GET /identity/v3/role_assignments?user.id=0e73fa44e2bc407b9d4825ea79bf107b&scope.project.id=99d46889c847422eafee000f4388dbed&role.id=22819414415942d399db534708fe7103 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:19:00.829207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020842a9-3642-4afd-acbe-572086e6d712 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:00.829763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020842a9-3642-4afd-acbe-572086e6d712 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:00.829763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020842a9-3642-4afd-acbe-572086e6d712 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:00.831132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 597/1186] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:19:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:19:00.837561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372600fa-417e-4a1c-b59a-10f7f7458774 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:00.838155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372600fa-417e-4a1c-b59a-10f7f7458774 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:00.838288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372600fa-417e-4a1c-b59a-10f7f7458774 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:00.869413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-372600fa-417e-4a1c-b59a-10f7f7458774 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:19:00.870646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-372600fa-417e-4a1c-b59a-10f7f7458774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:19:00.951752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 590/1187] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:19:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:19:01.225041 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020842a9-3642-4afd-acbe-572086e6d712 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:01.258748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:01.344831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020842a9-3642-4afd-acbe-572086e6d712 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:01.380355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7917c9-a05f-4a56-a4b6-2255fe65a422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:01.380976 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7917c9-a05f-4a56-a4b6-2255fe65a422 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:01.381249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7917c9-a05f-4a56-a4b6-2255fe65a422 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:01.381511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7917c9-a05f-4a56-a4b6-2255fe65a422 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:01.391791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7917c9-a05f-4a56-a4b6-2255fe65a422 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-dc7917c9-a05f-4a56-a4b6-2255fe65a422 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:19:01.396489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:19:01.397609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:19:01.398616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 598/1188] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:19:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:19:01.404669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372600fa-417e-4a1c-b59a-10f7f7458774 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:01.474430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:01.513279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372600fa-417e-4a1c-b59a-10f7f7458774 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:01.548965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8672fc-ae9f-4692-85ee-81c4da49a9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:01.549566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8672fc-ae9f-4692-85ee-81c4da49a9bd None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:01.549835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8672fc-ae9f-4692-85ee-81c4da49a9bd None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:01.550127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8672fc-ae9f-4692-85ee-81c4da49a9bd None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:01.550781 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8672fc-ae9f-4692-85ee-81c4da49a9bd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:19:01.551078 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8672fc-ae9f-4692-85ee-81c4da49a9bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:19:01.552443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8672fc-ae9f-4692-85ee-81c4da49a9bd None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:19:01.561055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 591/1189] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:19:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:19:01.568398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7917c9-a05f-4a56-a4b6-2255fe65a422 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:01.661273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:01.677256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7917c9-a05f-4a56-a4b6-2255fe65a422 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:01.721706 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6765acef-42c2-42f4-8c8b-f78248590e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:01.722681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765acef-42c2-42f4-8c8b-f78248590e36 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:01.723050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765acef-42c2-42f4-8c8b-f78248590e36 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:01.723401 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765acef-42c2-42f4-8c8b-f78248590e36 None admin] PATH_INFO: `/v3/users/0e73fa44e2bc407b9d4825ea79bf107b` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:01.725140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6765acef-42c2-42f4-8c8b-f78248590e36 None admin] RBAC: Authorizing `identity:get_user(user_id=0e73fa44e2bc407b9d4825ea79bf107b)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:19:01.727357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6765acef-42c2-42f4-8c8b-f78248590e36 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:19:01.729067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 599/1190] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:19:01 2022] GET /identity/v3/users/0e73fa44e2bc407b9d4825ea79bf107b => generated 311 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:19:01.736051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8672fc-ae9f-4692-85ee-81c4da49a9bd None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:01.849374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8672fc-ae9f-4692-85ee-81c4da49a9bd None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:01.885651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071a3a58-fc0f-4c60-8f9a-44fda9004e63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:01.886301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071a3a58-fc0f-4c60-8f9a-44fda9004e63 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:01.886576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071a3a58-fc0f-4c60-8f9a-44fda9004e63 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:01.886842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071a3a58-fc0f-4c60-8f9a-44fda9004e63 None admin] PATH_INFO: `/v3/projects/99d46889c847422eafee000f4388dbed` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:01.888018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071a3a58-fc0f-4c60-8f9a-44fda9004e63 None admin] RBAC: Authorizing `identity:get_project(project_id=99d46889c847422eafee000f4388dbed)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:19:01.889460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071a3a58-fc0f-4c60-8f9a-44fda9004e63 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:19:01.891116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 592/1191] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:19:01 2022] GET /identity/v3/projects/99d46889c847422eafee000f4388dbed => generated 361 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:19:01.898069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6765acef-42c2-42f4-8c8b-f78248590e36 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:02.005334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6765acef-42c2-42f4-8c8b-f78248590e36 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:02.041533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e03c62-cc60-4274-8876-c4a444f708b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:02.042209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e03c62-cc60-4274-8876-c4a444f708b8 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:02.042484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e03c62-cc60-4274-8876-c4a444f708b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:02.042756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e03c62-cc60-4274-8876-c4a444f708b8 None admin] PATH_INFO: `/v3/projects/99d46889c847422eafee000f4388dbed/users/0e73fa44e2bc407b9d4825ea79bf107b/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:02.044665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e03c62-cc60-4274-8876-c4a444f708b8 None admin] RBAC: Authorizing `identity:create_grant(project_id=99d46889c847422eafee000f4388dbed, user_id=0e73fa44e2bc407b9d4825ea79bf107b, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:19:02.046073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e03c62-cc60-4274-8876-c4a444f708b8 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:19:02.062924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 600/1192] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:19:01 2022] PUT /identity/v3/projects/99d46889c847422eafee000f4388dbed/users/0e73fa44e2bc407b9d4825ea79bf107b/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:19:04.212636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a1953-e41d-4a28-98cd-a04cf3832a9e None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:04.213537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a1953-e41d-4a28-98cd-a04cf3832a9e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:04.213537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a1953-e41d-4a28-98cd-a04cf3832a9e None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:04.215793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 593/1193] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:19:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:19:04.222259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37df943a-072e-4ae6-a594-e12bf6fef852 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:04.222963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37df943a-072e-4ae6-a594-e12bf6fef852 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:04.223125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37df943a-072e-4ae6-a594-e12bf6fef852 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:04.244589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-37df943a-072e-4ae6-a594-e12bf6fef852 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:19:04.245767 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-37df943a-072e-4ae6-a594-e12bf6fef852 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:19:04.255047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:04.320319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 601/1194] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:19:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:19:04.601218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a1953-e41d-4a28-98cd-a04cf3832a9e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:04.605579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:04.714686 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a1953-e41d-4a28-98cd-a04cf3832a9e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:04.750673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a984493-b570-4da3-b26a-b8284d81a22d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:04.751289 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a984493-b570-4da3-b26a-b8284d81a22d None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:04.751565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a984493-b570-4da3-b26a-b8284d81a22d None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:04.751833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a984493-b570-4da3-b26a-b8284d81a22d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:04.762999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a984493-b570-4da3-b26a-b8284d81a22d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-9a984493-b570-4da3-b26a-b8284d81a22d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:19:04.767717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:19:04.768770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:19:04.769737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 594/1195] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:19:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:19:04.775661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37df943a-072e-4ae6-a594-e12bf6fef852 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:04.892263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37df943a-072e-4ae6-a594-e12bf6fef852 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:04.929616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c22c91-b764-42be-a46f-361614de7b13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:04.930246 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c22c91-b764-42be-a46f-361614de7b13 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:04.930522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c22c91-b764-42be-a46f-361614de7b13 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:04.930784 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c22c91-b764-42be-a46f-361614de7b13 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:04.931434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c22c91-b764-42be-a46f-361614de7b13 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:19:04.931734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c22c91-b764-42be-a46f-361614de7b13 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:19:04.933103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c22c91-b764-42be-a46f-361614de7b13 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:19:04.941901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 602/1196] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:19:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:19:04.948058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a984493-b570-4da3-b26a-b8284d81a22d None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:05.012270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:05.057017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a984493-b570-4da3-b26a-b8284d81a22d None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:05.092241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caad7b7-1242-461f-af9f-b2a9c278d4da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:05.092889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caad7b7-1242-461f-af9f-b2a9c278d4da None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:05.093138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caad7b7-1242-461f-af9f-b2a9c278d4da None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:05.093407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caad7b7-1242-461f-af9f-b2a9c278d4da None admin] PATH_INFO: `/v3/users/0e73fa44e2bc407b9d4825ea79bf107b` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:05.094581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caad7b7-1242-461f-af9f-b2a9c278d4da None admin] RBAC: Authorizing `identity:get_user(user_id=0e73fa44e2bc407b9d4825ea79bf107b)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:19:05.096027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caad7b7-1242-461f-af9f-b2a9c278d4da None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:19:05.097604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 595/1197] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Sep 13 15:19:04 2022] GET /identity/v3/users/0e73fa44e2bc407b9d4825ea79bf107b => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:19:05.107086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c22c91-b764-42be-a46f-361614de7b13 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:05.121035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:05.222972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c22c91-b764-42be-a46f-361614de7b13 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:05.265705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d96ce02-a1be-4efa-94be-3db16e4af486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:05.266333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d96ce02-a1be-4efa-94be-3db16e4af486 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:05.266606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d96ce02-a1be-4efa-94be-3db16e4af486 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:05.266867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d96ce02-a1be-4efa-94be-3db16e4af486 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:05.350184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d96ce02-a1be-4efa-94be-3db16e4af486 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:19:05.352370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d96ce02-a1be-4efa-94be-3db16e4af486 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:19:05.365238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:05.462263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 603/1198] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:19:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:19:05.471178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caad7b7-1242-461f-af9f-b2a9c278d4da None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:05.576776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caad7b7-1242-461f-af9f-b2a9c278d4da None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:05.608285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:05.615319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb6ea12-8daf-4190-8d7c-31a2a0b7d2aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:05.615958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb6ea12-8daf-4190-8d7c-31a2a0b7d2aa None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:05.616217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb6ea12-8daf-4190-8d7c-31a2a0b7d2aa None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:05.616486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb6ea12-8daf-4190-8d7c-31a2a0b7d2aa None admin] PATH_INFO: `/v3/projects/99d46889c847422eafee000f4388dbed` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:05.617635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb6ea12-8daf-4190-8d7c-31a2a0b7d2aa None admin] RBAC: Authorizing `identity:get_project(project_id=99d46889c847422eafee000f4388dbed)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:19:05.619614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb6ea12-8daf-4190-8d7c-31a2a0b7d2aa None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:19:05.621230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 596/1199] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Sep 13 15:19:05 2022] GET /identity/v3/projects/99d46889c847422eafee000f4388dbed => generated 361 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:19:05.627577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d96ce02-a1be-4efa-94be-3db16e4af486 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:19:05.724876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d96ce02-a1be-4efa-94be-3db16e4af486 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:19:05.736663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:05.766888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5024f28e-f41d-412c-a8c6-942b839bdf55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:19:05.767479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5024f28e-f41d-412c-a8c6-942b839bdf55 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:19:05.767750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5024f28e-f41d-412c-a8c6-942b839bdf55 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:19:05.768011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5024f28e-f41d-412c-a8c6-942b839bdf55 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:19:05.768724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5024f28e-f41d-412c-a8c6-942b839bdf55 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=99d46889c847422eafee000f4388dbed, user.id=0e73fa44e2bc407b9d4825ea79bf107b) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:19:05.769030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5024f28e-f41d-412c-a8c6-942b839bdf55 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:19:05.770477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5024f28e-f41d-412c-a8c6-942b839bdf55 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:19:05.776245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 604/1200] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:19:05 2022] GET /identity/v3/role_assignments?user.id=0e73fa44e2bc407b9d4825ea79bf107b&scope.project.id=99d46889c847422eafee000f4388dbed&role.id=22819414415942d399db534708fe7103 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:19:15.613764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:19:15.773517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=15 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:02.135521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413ed7b3-955e-478e-a608-45a90ca22def None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:02.135521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413ed7b3-955e-478e-a608-45a90ca22def None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:02.135521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413ed7b3-955e-478e-a608-45a90ca22def None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:02.135521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 597/1201] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:02.142927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eca0b7-ae80-4977-a864-42fc26a0b09e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:02.143449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eca0b7-ae80-4977-a864-42fc26a0b09e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:02.143449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eca0b7-ae80-4977-a864-42fc26a0b09e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:02.168080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-77eca0b7-ae80-4977-a864-42fc26a0b09e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:02.169269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-77eca0b7-ae80-4977-a864-42fc26a0b09e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:02.251299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 605/1202] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:02.542897 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413ed7b3-955e-478e-a608-45a90ca22def None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:02.668759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413ed7b3-955e-478e-a608-45a90ca22def None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:02.712460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85b09cf-0823-468d-b8aa-bdf12c8cb418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:02.713569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85b09cf-0823-468d-b8aa-bdf12c8cb418 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:02.714053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85b09cf-0823-468d-b8aa-bdf12c8cb418 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:02.714544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85b09cf-0823-468d-b8aa-bdf12c8cb418 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:02.727625 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85b09cf-0823-468d-b8aa-bdf12c8cb418 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:02.729107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85b09cf-0823-468d-b8aa-bdf12c8cb418 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-e85b09cf-0823-468d-b8aa-bdf12c8cb418 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:02.746218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:02.747336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:02.748309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:02.748309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:02.748309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:02.752505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 598/1203] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:02.754126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eca0b7-ae80-4977-a864-42fc26a0b09e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:02.877122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eca0b7-ae80-4977-a864-42fc26a0b09e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:02.915589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:02.924267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a15c9c-ad26-4360-ad27-7588ae10ac12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:02.925052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a15c9c-ad26-4360-ad27-7588ae10ac12 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:02.925358 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a15c9c-ad26-4360-ad27-7588ae10ac12 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:02.925640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a15c9c-ad26-4360-ad27-7588ae10ac12 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:02.926631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a15c9c-ad26-4360-ad27-7588ae10ac12 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:02.927167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a15c9c-ad26-4360-ad27-7588ae10ac12 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:02.929552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a15c9c-ad26-4360-ad27-7588ae10ac12 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:02.942740 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 606/1204] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:02.953988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85b09cf-0823-468d-b8aa-bdf12c8cb418 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:03.018667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:03.085309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85b09cf-0823-468d-b8aa-bdf12c8cb418 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:03.150539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9f6569-55ee-445e-a2c8-038d6317d9c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:03.151317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9f6569-55ee-445e-a2c8-038d6317d9c9 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:03.151608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9f6569-55ee-445e-a2c8-038d6317d9c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:03.151980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9f6569-55ee-445e-a2c8-038d6317d9c9 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:03.152725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9f6569-55ee-445e-a2c8-038d6317d9c9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:03.154662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9f6569-55ee-445e-a2c8-038d6317d9c9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:03.184835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 599/1205] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Sep 13 15:20:02 2022] POST /identity/v3/users => generated 251 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:20:05.638129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3602d1-857e-43f4-95b7-7470ec3dc7e2 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:05.638757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3602d1-857e-43f4-95b7-7470ec3dc7e2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:05.638757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3602d1-857e-43f4-95b7-7470ec3dc7e2 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:05.642276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 607/1206] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:05.646074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2817e6b8-dd47-47f4-853a-dbe429719455 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:05.646487 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2817e6b8-dd47-47f4-853a-dbe429719455 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:05.646900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2817e6b8-dd47-47f4-853a-dbe429719455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:05.669179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2817e6b8-dd47-47f4-853a-dbe429719455 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:05.670543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2817e6b8-dd47-47f4-853a-dbe429719455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:05.752233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 600/1207] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:06.042705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3602d1-857e-43f4-95b7-7470ec3dc7e2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:06.159940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3602d1-857e-43f4-95b7-7470ec3dc7e2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:06.200320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540d0d15-3a26-4268-a5c3-db170e25d46f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:06.200953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540d0d15-3a26-4268-a5c3-db170e25d46f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:06.201232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540d0d15-3a26-4268-a5c3-db170e25d46f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:06.201559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540d0d15-3a26-4268-a5c3-db170e25d46f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:06.212885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540d0d15-3a26-4268-a5c3-db170e25d46f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-540d0d15-3a26-4268-a5c3-db170e25d46f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:20:06.217562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:06.218637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:06.219559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 608/1208] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:20:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:06.225617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2817e6b8-dd47-47f4-853a-dbe429719455 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:06.269325 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:06.341997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2817e6b8-dd47-47f4-853a-dbe429719455 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:06.390412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72074ecf-de6f-4ead-bc46-e196de2cea10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:06.391110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72074ecf-de6f-4ead-bc46-e196de2cea10 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:06.391383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72074ecf-de6f-4ead-bc46-e196de2cea10 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:06.391675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72074ecf-de6f-4ead-bc46-e196de2cea10 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:06.392370 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72074ecf-de6f-4ead-bc46-e196de2cea10 None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:06.392678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72074ecf-de6f-4ead-bc46-e196de2cea10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:06.394176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72074ecf-de6f-4ead-bc46-e196de2cea10 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:06.406641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 601/1209] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:20:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:06.413445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540d0d15-3a26-4268-a5c3-db170e25d46f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:06.477162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:06.534025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540d0d15-3a26-4268-a5c3-db170e25d46f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:06.575931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c3f37d-dbd1-42a5-964e-187e07882b6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:06.576598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c3f37d-dbd1-42a5-964e-187e07882b6d None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:06.576894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c3f37d-dbd1-42a5-964e-187e07882b6d None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:06.577183 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c3f37d-dbd1-42a5-964e-187e07882b6d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:06.591016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c3f37d-dbd1-42a5-964e-187e07882b6d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:06.592685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c3f37d-dbd1-42a5-964e-187e07882b6d None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-50c3f37d-dbd1-42a5-964e-187e07882b6d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:06.609876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:06.612663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:06.613591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:06.613591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:06.613591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:06.613591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 609/1210] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:06.617927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72074ecf-de6f-4ead-bc46-e196de2cea10 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:06.758053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:06.761769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72074ecf-de6f-4ead-bc46-e196de2cea10 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:06.799430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f073a1-d480-4dce-b4d6-478581f777b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:06.800056 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f073a1-d480-4dce-b4d6-478581f777b2 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:06.800333 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f073a1-d480-4dce-b4d6-478581f777b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:06.800628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f073a1-d480-4dce-b4d6-478581f777b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:06.801418 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f073a1-d480-4dce-b4d6-478581f777b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:06.801716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f073a1-d480-4dce-b4d6-478581f777b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:06.803258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f073a1-d480-4dce-b4d6-478581f777b2 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:06.815972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 602/1211] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:06.823337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c3f37d-dbd1-42a5-964e-187e07882b6d None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:06.938050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c3f37d-dbd1-42a5-964e-187e07882b6d None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:06.939861 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:06.978430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797af8e1-7ec7-49d2-91a2-5399833bbc38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:06.978901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797af8e1-7ec7-49d2-91a2-5399833bbc38 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:06.979184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797af8e1-7ec7-49d2-91a2-5399833bbc38 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:06.979488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797af8e1-7ec7-49d2-91a2-5399833bbc38 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:06.988860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797af8e1-7ec7-49d2-91a2-5399833bbc38 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:06.990435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797af8e1-7ec7-49d2-91a2-5399833bbc38 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-797af8e1-7ec7-49d2-91a2-5399833bbc38 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:20:07.003533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:07.004826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:07.006058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 610/1212] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:06 2022] GET /identity/v3/users/placement => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:07.011063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f073a1-d480-4dce-b4d6-478581f777b2 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:07.131791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f073a1-d480-4dce-b4d6-478581f777b2 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:07.173706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d08b671-a243-413a-8110-965568acd7b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:07.174629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d08b671-a243-413a-8110-965568acd7b8 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:07.175044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d08b671-a243-413a-8110-965568acd7b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:07.175318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d08b671-a243-413a-8110-965568acd7b8 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:07.176209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d08b671-a243-413a-8110-965568acd7b8 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:07.176611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d08b671-a243-413a-8110-965568acd7b8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:07.178358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d08b671-a243-413a-8110-965568acd7b8 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:07.198595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 603/1213] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Sep 13 15:20:07 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:07.205482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797af8e1-7ec7-49d2-91a2-5399833bbc38 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:07.321069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797af8e1-7ec7-49d2-91a2-5399833bbc38 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:07.364416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed81430a-c34e-4ab9-a247-6ea7b46993cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:07.365117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed81430a-c34e-4ab9-a247-6ea7b46993cc None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:07.365391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed81430a-c34e-4ab9-a247-6ea7b46993cc None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:07.365662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed81430a-c34e-4ab9-a247-6ea7b46993cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:07.398002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:07.448952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed81430a-c34e-4ab9-a247-6ea7b46993cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:07.450552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed81430a-c34e-4ab9-a247-6ea7b46993cc None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:07.578736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 611/1214] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:20:07 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:20:07.585953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d08b671-a243-413a-8110-965568acd7b8 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:07.609155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:07.694329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d08b671-a243-413a-8110-965568acd7b8 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:07.728951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33ae590-269f-48b5-881e-b40446272886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:07.729532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33ae590-269f-48b5-881e-b40446272886 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:07.729784 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33ae590-269f-48b5-881e-b40446272886 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:07.730051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33ae590-269f-48b5-881e-b40446272886 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:07.740798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33ae590-269f-48b5-881e-b40446272886 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:07.742287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33ae590-269f-48b5-881e-b40446272886 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-c33ae590-269f-48b5-881e-b40446272886 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:07.759143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:07.760668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:07.761585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:07.761585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:07.761585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:07.761585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 604/1215] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:07.766354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed81430a-c34e-4ab9-a247-6ea7b46993cc None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:07.832228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:07.873989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed81430a-c34e-4ab9-a247-6ea7b46993cc None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:07.910883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23deb08c-f5c6-41ae-ac20-51f157e210ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:07.911541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23deb08c-f5c6-41ae-ac20-51f157e210ec None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:07.911834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23deb08c-f5c6-41ae-ac20-51f157e210ec None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:07.912104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23deb08c-f5c6-41ae-ac20-51f157e210ec None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:07.912730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23deb08c-f5c6-41ae-ac20-51f157e210ec None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:07.913034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23deb08c-f5c6-41ae-ac20-51f157e210ec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:07.914433 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23deb08c-f5c6-41ae-ac20-51f157e210ec None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:07.927019 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 612/1216] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:07.933155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33ae590-269f-48b5-881e-b40446272886 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:08.016065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:08.053385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33ae590-269f-48b5-881e-b40446272886 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:08.099676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c928b54-5fec-49f0-a6ff-71c58e5b1690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:08.100310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c928b54-5fec-49f0-a6ff-71c58e5b1690 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:08.100582 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c928b54-5fec-49f0-a6ff-71c58e5b1690 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:08.100850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c928b54-5fec-49f0-a6ff-71c58e5b1690 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:08.108212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c928b54-5fec-49f0-a6ff-71c58e5b1690 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:08.109675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c928b54-5fec-49f0-a6ff-71c58e5b1690 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-6c928b54-5fec-49f0-a6ff-71c58e5b1690 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:20:08.121884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:08.123303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:08.124201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:20:08.124201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:20:08.124201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:20:08.124201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:20:08.124201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:20:08.124201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:20:08.124201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:08.124201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:08.124201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 605/1217] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:20:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:08.129436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23deb08c-f5c6-41ae-ac20-51f157e210ec None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:08.235332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23deb08c-f5c6-41ae-ac20-51f157e210ec None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:08.260621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:08.276092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b829e5fd-8561-4e01-9bf0-b0e108cf800f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:08.276674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b829e5fd-8561-4e01-9bf0-b0e108cf800f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:08.276945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b829e5fd-8561-4e01-9bf0-b0e108cf800f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:08.277207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b829e5fd-8561-4e01-9bf0-b0e108cf800f None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:08.278068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b829e5fd-8561-4e01-9bf0-b0e108cf800f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:08.278316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b829e5fd-8561-4e01-9bf0-b0e108cf800f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:08.279959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b829e5fd-8561-4e01-9bf0-b0e108cf800f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:08.294203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 613/1218] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:20:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:08.300767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c928b54-5fec-49f0-a6ff-71c58e5b1690 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:08.410251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c928b54-5fec-49f0-a6ff-71c58e5b1690 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:08.448106 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:08.454551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e1c474-df98-42c1-a570-bbf24e8f362e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:08.455237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e1c474-df98-42c1-a570-bbf24e8f362e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:08.455570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e1c474-df98-42c1-a570-bbf24e8f362e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:08.455893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e1c474-df98-42c1-a570-bbf24e8f362e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:08.456659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e1c474-df98-42c1-a570-bbf24e8f362e None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=4bc6be7644b24ca9bab23128b41a7148) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:08.457019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e1c474-df98-42c1-a570-bbf24e8f362e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:08.458567 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e1c474-df98-42c1-a570-bbf24e8f362e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:08.474148 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 606/1219] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:20:08 2022] GET /identity/v3/role_assignments?user.id=4bc6be7644b24ca9bab23128b41a7148&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:10.707313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b6dd10-8d11-49d1-a154-cfd7e4b9bab7 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:10.707881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b6dd10-8d11-49d1-a154-cfd7e4b9bab7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:10.707881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b6dd10-8d11-49d1-a154-cfd7e4b9bab7 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:10.709258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 614/1220] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:10.714905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955390fb-03c2-4254-928e-131bfb6b312d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:10.715388 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955390fb-03c2-4254-928e-131bfb6b312d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:10.715512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955390fb-03c2-4254-928e-131bfb6b312d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:10.735099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-955390fb-03c2-4254-928e-131bfb6b312d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:10.736312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-955390fb-03c2-4254-928e-131bfb6b312d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:10.807650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 607/1221] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:11.076653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b6dd10-8d11-49d1-a154-cfd7e4b9bab7 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:11.183884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b6dd10-8d11-49d1-a154-cfd7e4b9bab7 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:11.219704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307cc3a8-97ce-4a58-8c40-1d9be1e1be7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:11.220334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307cc3a8-97ce-4a58-8c40-1d9be1e1be7f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:11.220627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307cc3a8-97ce-4a58-8c40-1d9be1e1be7f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:11.220873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307cc3a8-97ce-4a58-8c40-1d9be1e1be7f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:11.231005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307cc3a8-97ce-4a58-8c40-1d9be1e1be7f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-307cc3a8-97ce-4a58-8c40-1d9be1e1be7f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:20:11.236018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:11.237192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:11.238475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 615/1222] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:20:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:20:11.243442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955390fb-03c2-4254-928e-131bfb6b312d None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:11.364428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955390fb-03c2-4254-928e-131bfb6b312d None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:11.392575 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:11.404555 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5bc63e-7b3e-4254-8a4a-56df7d15312c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:11.405178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5bc63e-7b3e-4254-8a4a-56df7d15312c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:11.405455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5bc63e-7b3e-4254-8a4a-56df7d15312c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:11.405723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5bc63e-7b3e-4254-8a4a-56df7d15312c None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:11.406424 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5bc63e-7b3e-4254-8a4a-56df7d15312c None admin] RBAC: Adding query filter params (name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:11.406722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5bc63e-7b3e-4254-8a4a-56df7d15312c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:11.408115 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5bc63e-7b3e-4254-8a4a-56df7d15312c None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:11.417981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 608/1223] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:20:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:11.424303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307cc3a8-97ce-4a58-8c40-1d9be1e1be7f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:11.472871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:11.537334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307cc3a8-97ce-4a58-8c40-1d9be1e1be7f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:11.575698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfd9f86-f118-43c7-a01b-51ce944e3281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:11.576456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfd9f86-f118-43c7-a01b-51ce944e3281 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:11.576729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfd9f86-f118-43c7-a01b-51ce944e3281 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:11.577001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfd9f86-f118-43c7-a01b-51ce944e3281 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:11.587545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dfd9f86-f118-43c7-a01b-51ce944e3281 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:11.589143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dfd9f86-f118-43c7-a01b-51ce944e3281 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-6dfd9f86-f118-43c7-a01b-51ce944e3281 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:11.605325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:11.606981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:11.608334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:11.608334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:11.608334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:11.608334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 616/1224] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:11.613396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5bc63e-7b3e-4254-8a4a-56df7d15312c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:11.726936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5bc63e-7b3e-4254-8a4a-56df7d15312c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:11.763403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7331be3-5ff9-4df3-bcce-f6a90bbb4b70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:11.764062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7331be3-5ff9-4df3-bcce-f6a90bbb4b70 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:11.764406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7331be3-5ff9-4df3-bcce-f6a90bbb4b70 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:11.764679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7331be3-5ff9-4df3-bcce-f6a90bbb4b70 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:11.765361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7331be3-5ff9-4df3-bcce-f6a90bbb4b70 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:11.765654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7331be3-5ff9-4df3-bcce-f6a90bbb4b70 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:11.767302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7331be3-5ff9-4df3-bcce-f6a90bbb4b70 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:11.782656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 609/1225] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:11.788462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfd9f86-f118-43c7-a01b-51ce944e3281 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:11.889458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:11.897256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfd9f86-f118-43c7-a01b-51ce944e3281 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:11.935562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9c4465-da59-4f9b-95c7-4440303a571f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:11.936268 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9c4465-da59-4f9b-95c7-4440303a571f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:11.936527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9c4465-da59-4f9b-95c7-4440303a571f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:11.936837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9c4465-da59-4f9b-95c7-4440303a571f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:11.952940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9c4465-da59-4f9b-95c7-4440303a571f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:11.954676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9c4465-da59-4f9b-95c7-4440303a571f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-dc9c4465-da59-4f9b-95c7-4440303a571f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:20:11.968190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:11.973030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:11.974495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 617/1226] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:11 2022] GET /identity/v3/users/placement => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:11.975995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7331be3-5ff9-4df3-bcce-f6a90bbb4b70 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:12.027634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:12.087479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7331be3-5ff9-4df3-bcce-f6a90bbb4b70 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:12.123865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ed74cd-28a5-4d8a-b953-80e5221c18e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:12.124575 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ed74cd-28a5-4d8a-b953-80e5221c18e5 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:12.124892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ed74cd-28a5-4d8a-b953-80e5221c18e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:12.125169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ed74cd-28a5-4d8a-b953-80e5221c18e5 None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:12.125997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ed74cd-28a5-4d8a-b953-80e5221c18e5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:12.126364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ed74cd-28a5-4d8a-b953-80e5221c18e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:12.128095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ed74cd-28a5-4d8a-b953-80e5221c18e5 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:12.148425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 610/1227] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Sep 13 15:20:11 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:12.154746 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9c4465-da59-4f9b-95c7-4440303a571f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:12.261099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9c4465-da59-4f9b-95c7-4440303a571f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:12.296993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd75017f-dda7-435e-8802-32c5546ec9e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:12.297928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd75017f-dda7-435e-8802-32c5546ec9e4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:12.298234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd75017f-dda7-435e-8802-32c5546ec9e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:12.298499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd75017f-dda7-435e-8802-32c5546ec9e4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:12.310071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd75017f-dda7-435e-8802-32c5546ec9e4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:12.311511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd75017f-dda7-435e-8802-32c5546ec9e4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-cd75017f-dda7-435e-8802-32c5546ec9e4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:12.328127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:12.334779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:12.335807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:12.335807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:12.335807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:12.335807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 618/1228] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:12.336064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ed74cd-28a5-4d8a-b953-80e5221c18e5 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:12.450525 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ed74cd-28a5-4d8a-b953-80e5221c18e5 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:12.453926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:12.486542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94ec024-3de1-42c5-91e5-4d090cd648bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:12.487194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ec024-3de1-42c5-91e5-4d090cd648bb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:12.487468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ec024-3de1-42c5-91e5-4d090cd648bb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:12.487760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94ec024-3de1-42c5-91e5-4d090cd648bb None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:12.488365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94ec024-3de1-42c5-91e5-4d090cd648bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:12.488707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94ec024-3de1-42c5-91e5-4d090cd648bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:12.490451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94ec024-3de1-42c5-91e5-4d090cd648bb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:12.502376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 611/1229] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:12.508494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd75017f-dda7-435e-8802-32c5546ec9e4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:12.518168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:12.610188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd75017f-dda7-435e-8802-32c5546ec9e4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:12.652298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff13056-85d8-4a57-9df6-2d8b593b77e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:12.652943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff13056-85d8-4a57-9df6-2d8b593b77e0 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:12.653217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff13056-85d8-4a57-9df6-2d8b593b77e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:12.653479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff13056-85d8-4a57-9df6-2d8b593b77e0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:12.660283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff13056-85d8-4a57-9df6-2d8b593b77e0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:12.661737 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff13056-85d8-4a57-9df6-2d8b593b77e0 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-dff13056-85d8-4a57-9df6-2d8b593b77e0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:20:12.673162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:12.674242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:12.675460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:20:12.675460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:20:12.675460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:20:12.675460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:20:12.675460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:20:12.675460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:20:12.675460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:12.675460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:12.675460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 619/1230] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:20:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:20:12.680900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94ec024-3de1-42c5-91e5-4d090cd648bb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:12.783328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94ec024-3de1-42c5-91e5-4d090cd648bb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:12.818637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80a57b2-0d5f-472d-9605-adc121e1c308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:12.819000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80a57b2-0d5f-472d-9605-adc121e1c308 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:12.819227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80a57b2-0d5f-472d-9605-adc121e1c308 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:12.819495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80a57b2-0d5f-472d-9605-adc121e1c308 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:12.820179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80a57b2-0d5f-472d-9605-adc121e1c308 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:12.820563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80a57b2-0d5f-472d-9605-adc121e1c308 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:12.822153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80a57b2-0d5f-472d-9605-adc121e1c308 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:12.837328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 612/1231] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:20:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:20:12.843828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff13056-85d8-4a57-9df6-2d8b593b77e0 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:12.917156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:12.955856 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff13056-85d8-4a57-9df6-2d8b593b77e0 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:12.991909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d82c9cf-3ccf-4607-980c-b4f5348e7b56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:12.992556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d82c9cf-3ccf-4607-980c-b4f5348e7b56 None admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:12.992881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d82c9cf-3ccf-4607-980c-b4f5348e7b56 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:12.993175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d82c9cf-3ccf-4607-980c-b4f5348e7b56 None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/4bc6be7644b24ca9bab23128b41a7148/roles/da0ea75123ba496fb410e87b8df992ba` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:13.008090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d82c9cf-3ccf-4607-980c-b4f5348e7b56 None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=4bc6be7644b24ca9bab23128b41a7148, role_id=da0ea75123ba496fb410e87b8df992ba)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:13.009521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d82c9cf-3ccf-4607-980c-b4f5348e7b56 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:13.021161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 620/1232] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:20:12 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/4bc6be7644b24ca9bab23128b41a7148/roles/da0ea75123ba496fb410e87b8df992ba => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:20:15.354684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d7bb4d-5e18-4a38-94ed-c085e8815552 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:15.355265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d7bb4d-5e18-4a38-94ed-c085e8815552 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:15.355444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d7bb4d-5e18-4a38-94ed-c085e8815552 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:15.357372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 613/1233] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:15.364297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8289467a-379f-47f6-a76a-312b9e0b34bb None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:15.364853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8289467a-379f-47f6-a76a-312b9e0b34bb None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:15.364853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8289467a-379f-47f6-a76a-312b9e0b34bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:15.385087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8289467a-379f-47f6-a76a-312b9e0b34bb None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:15.386437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8289467a-379f-47f6-a76a-312b9e0b34bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:15.457660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 621/1234] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:15.752699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d7bb4d-5e18-4a38-94ed-c085e8815552 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:15.779664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:15.868446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d7bb4d-5e18-4a38-94ed-c085e8815552 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:15.905930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d687449c-1f61-45ba-ad47-495d3cb85b19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:15.906611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687449c-1f61-45ba-ad47-495d3cb85b19 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:15.906893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687449c-1f61-45ba-ad47-495d3cb85b19 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:15.907166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687449c-1f61-45ba-ad47-495d3cb85b19 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:15.918254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d687449c-1f61-45ba-ad47-495d3cb85b19 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-d687449c-1f61-45ba-ad47-495d3cb85b19 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:20:15.923009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:15.924117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:15.924972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 614/1235] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Sep 13 15:20:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:15.930735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8289467a-379f-47f6-a76a-312b9e0b34bb None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:15.980650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:16.047104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8289467a-379f-47f6-a76a-312b9e0b34bb None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:16.084108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c723005-7f29-4804-b3ed-0ddd5340dd51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:16.084734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c723005-7f29-4804-b3ed-0ddd5340dd51 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:16.085016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c723005-7f29-4804-b3ed-0ddd5340dd51 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:16.085289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c723005-7f29-4804-b3ed-0ddd5340dd51 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:16.085983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c723005-7f29-4804-b3ed-0ddd5340dd51 None admin] RBAC: Adding query filter params (name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:16.086295 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c723005-7f29-4804-b3ed-0ddd5340dd51 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:16.087676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c723005-7f29-4804-b3ed-0ddd5340dd51 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:16.101317 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 622/1236] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Sep 13 15:20:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:16.102626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d687449c-1f61-45ba-ad47-495d3cb85b19 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:16.201945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:16.220702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d687449c-1f61-45ba-ad47-495d3cb85b19 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:16.258718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0899ef7a-3f0f-4b86-a584-8140806ac32e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:16.259427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0899ef7a-3f0f-4b86-a584-8140806ac32e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:16.259788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0899ef7a-3f0f-4b86-a584-8140806ac32e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:16.260065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0899ef7a-3f0f-4b86-a584-8140806ac32e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:16.270377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0899ef7a-3f0f-4b86-a584-8140806ac32e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:16.271923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0899ef7a-3f0f-4b86-a584-8140806ac32e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-0899ef7a-3f0f-4b86-a584-8140806ac32e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:16.286616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:16.287777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:16.288841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:16.288841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:16.288841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:16.288841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 615/1237] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:20:16.294251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c723005-7f29-4804-b3ed-0ddd5340dd51 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:16.411451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c723005-7f29-4804-b3ed-0ddd5340dd51 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:16.420617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:16.448335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d299d5a8-e536-4677-b93f-38d79cfd749e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:16.448945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d299d5a8-e536-4677-b93f-38d79cfd749e None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:16.449308 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d299d5a8-e536-4677-b93f-38d79cfd749e None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:16.449612 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d299d5a8-e536-4677-b93f-38d79cfd749e None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:16.450477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d299d5a8-e536-4677-b93f-38d79cfd749e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:16.450836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d299d5a8-e536-4677-b93f-38d79cfd749e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:16.452615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d299d5a8-e536-4677-b93f-38d79cfd749e None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:16.466171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 623/1238] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:16.472953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0899ef7a-3f0f-4b86-a584-8140806ac32e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:16.592948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0899ef7a-3f0f-4b86-a584-8140806ac32e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:16.635380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069c5eb3-a83a-432a-970a-70c87c29a71f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:16.636034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069c5eb3-a83a-432a-970a-70c87c29a71f None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:16.636322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069c5eb3-a83a-432a-970a-70c87c29a71f None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:16.636597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069c5eb3-a83a-432a-970a-70c87c29a71f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:16.639693 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:16.648955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069c5eb3-a83a-432a-970a-70c87c29a71f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:16.650519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069c5eb3-a83a-432a-970a-70c87c29a71f None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-069c5eb3-a83a-432a-970a-70c87c29a71f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:20:16.664594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:16.665807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:16.667194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 616/1239] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:16 2022] GET /identity/v3/users/placement => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:20:16.672225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d299d5a8-e536-4677-b93f-38d79cfd749e None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:16.790702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d299d5a8-e536-4677-b93f-38d79cfd749e None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:16.829695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07702115-86c5-4ffe-b771-1023cd870421 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:16.830377 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07702115-86c5-4ffe-b771-1023cd870421 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:16.830653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07702115-86c5-4ffe-b771-1023cd870421 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:16.830912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07702115-86c5-4ffe-b771-1023cd870421 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:16.831651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07702115-86c5-4ffe-b771-1023cd870421 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:16.831959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07702115-86c5-4ffe-b771-1023cd870421 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:16.833469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07702115-86c5-4ffe-b771-1023cd870421 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:16.859510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 624/1240] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Sep 13 15:20:16 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:16.860894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069c5eb3-a83a-432a-970a-70c87c29a71f None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:16.970272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069c5eb3-a83a-432a-970a-70c87c29a71f None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:17.008707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4625feea-9189-4836-ad13-7804a42d4ab1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:17.009446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4625feea-9189-4836-ad13-7804a42d4ab1 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:17.009771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4625feea-9189-4836-ad13-7804a42d4ab1 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:17.010038 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4625feea-9189-4836-ad13-7804a42d4ab1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:17.079265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:17.086871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4625feea-9189-4836-ad13-7804a42d4ab1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:17.088351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4625feea-9189-4836-ad13-7804a42d4ab1 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:17.193125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 617/1241] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:20:16 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:20:17.199156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07702115-86c5-4ffe-b771-1023cd870421 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:17.229362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:17.312574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07702115-86c5-4ffe-b771-1023cd870421 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:17.360969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c257cf-5e7d-4294-898e-60ad3cbd6b5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:17.362351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c257cf-5e7d-4294-898e-60ad3cbd6b5f None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:17.362805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c257cf-5e7d-4294-898e-60ad3cbd6b5f None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:17.363084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c257cf-5e7d-4294-898e-60ad3cbd6b5f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:17.375449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c257cf-5e7d-4294-898e-60ad3cbd6b5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:17.376926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c257cf-5e7d-4294-898e-60ad3cbd6b5f None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-26c257cf-5e7d-4294-898e-60ad3cbd6b5f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:17.396814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:17.397937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:17.398873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:17.398873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:17.398873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:17.398873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 625/1242] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:20:17.403883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4625feea-9189-4836-ad13-7804a42d4ab1 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:17.507894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:17.515066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4625feea-9189-4836-ad13-7804a42d4ab1 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:17.551568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed7b5e0-d1bf-49d6-945e-f9cc4d782b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:17.552223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed7b5e0-d1bf-49d6-945e-f9cc4d782b27 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:17.552499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed7b5e0-d1bf-49d6-945e-f9cc4d782b27 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:17.552764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed7b5e0-d1bf-49d6-945e-f9cc4d782b27 None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:17.553396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed7b5e0-d1bf-49d6-945e-f9cc4d782b27 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:17.553731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed7b5e0-d1bf-49d6-945e-f9cc4d782b27 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:17.555589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed7b5e0-d1bf-49d6-945e-f9cc4d782b27 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:17.567265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 618/1243] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:17.574582 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c257cf-5e7d-4294-898e-60ad3cbd6b5f None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:17.681618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:17.710436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c257cf-5e7d-4294-898e-60ad3cbd6b5f None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:17.752116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129c37b2-ecea-47f1-9bc9-7e8fb2a30e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:17.752857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c37b2-ecea-47f1-9bc9-7e8fb2a30e39 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:17.753318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c37b2-ecea-47f1-9bc9-7e8fb2a30e39 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:17.753599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c37b2-ecea-47f1-9bc9-7e8fb2a30e39 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:17.761918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129c37b2-ecea-47f1-9bc9-7e8fb2a30e39 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:17.763618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129c37b2-ecea-47f1-9bc9-7e8fb2a30e39 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-129c37b2-ecea-47f1-9bc9-7e8fb2a30e39 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:20:17.776397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:17.777355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:17.778427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:20:17.778427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:20:17.778427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:20:17.778427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:20:17.778427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:20:17.778427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:20:17.778427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:17.778427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:17.778427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 626/1244] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:20:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:17.784062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed7b5e0-d1bf-49d6-945e-f9cc4d782b27 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:17.891664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed7b5e0-d1bf-49d6-945e-f9cc4d782b27 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:17.929969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3c74df-5e7d-4695-8cdd-bd44020e50da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:17.930625 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c74df-5e7d-4695-8cdd-bd44020e50da None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:17.930912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c74df-5e7d-4695-8cdd-bd44020e50da None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:17.931186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3c74df-5e7d-4695-8cdd-bd44020e50da None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:17.931884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c74df-5e7d-4695-8cdd-bd44020e50da None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:17.932197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c74df-5e7d-4695-8cdd-bd44020e50da None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:17.933654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3c74df-5e7d-4695-8cdd-bd44020e50da None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:17.952282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 619/1245] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:20:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:20:17.953614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129c37b2-ecea-47f1-9bc9-7e8fb2a30e39 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:18.082318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129c37b2-ecea-47f1-9bc9-7e8fb2a30e39 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:18.120079 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf46a5d-71af-4bf9-b409-e845d968b2f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:18.120695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf46a5d-71af-4bf9-b409-e845d968b2f2 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:18.120966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf46a5d-71af-4bf9-b409-e845d968b2f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:18.121353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf46a5d-71af-4bf9-b409-e845d968b2f2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:18.122291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf46a5d-71af-4bf9-b409-e845d968b2f2 None admin] RBAC: Adding query filter params (role.id=da0ea75123ba496fb410e87b8df992ba, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=4bc6be7644b24ca9bab23128b41a7148) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:18.122597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf46a5d-71af-4bf9-b409-e845d968b2f2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:18.124091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf46a5d-71af-4bf9-b409-e845d968b2f2 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:18.131391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 627/1246] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:20:17 2022] GET /identity/v3/role_assignments?user.id=4bc6be7644b24ca9bab23128b41a7148&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=da0ea75123ba496fb410e87b8df992ba => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:20.654938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45edd36c-1e58-4d36-85e2-38baf5da1c1e None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:20.655496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45edd36c-1e58-4d36-85e2-38baf5da1c1e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:20.655496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45edd36c-1e58-4d36-85e2-38baf5da1c1e None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:20.657183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 620/1247] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:20.664698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82365be2-8e8f-4b57-85ae-f11a0811db90 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:20.665167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82365be2-8e8f-4b57-85ae-f11a0811db90 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:20.665260 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82365be2-8e8f-4b57-85ae-f11a0811db90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:20.673636 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:20.687576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-82365be2-8e8f-4b57-85ae-f11a0811db90 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:20.688858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-82365be2-8e8f-4b57-85ae-f11a0811db90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:20.774150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 628/1248] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:21.036112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45edd36c-1e58-4d36-85e2-38baf5da1c1e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:21.056831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:21.153861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45edd36c-1e58-4d36-85e2-38baf5da1c1e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:21.192369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6504d5-64b1-4579-adbd-259d30921633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:21.193026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6504d5-64b1-4579-adbd-259d30921633 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:21.193286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6504d5-64b1-4579-adbd-259d30921633 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:21.193582 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6504d5-64b1-4579-adbd-259d30921633 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:21.205219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6504d5-64b1-4579-adbd-259d30921633 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-ff6504d5-64b1-4579-adbd-259d30921633 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:20:21.210402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:21.211518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:21.212586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 621/1249] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:20:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:21.218312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82365be2-8e8f-4b57-85ae-f11a0811db90 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:21.324648 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82365be2-8e8f-4b57-85ae-f11a0811db90 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:21.363364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:21.369629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986e946b-2368-4501-a154-d501ea8373a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:21.370524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e946b-2368-4501-a154-d501ea8373a3 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:21.370818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e946b-2368-4501-a154-d501ea8373a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:21.371300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e946b-2368-4501-a154-d501ea8373a3 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:21.371982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986e946b-2368-4501-a154-d501ea8373a3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:21.372305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986e946b-2368-4501-a154-d501ea8373a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:21.373917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986e946b-2368-4501-a154-d501ea8373a3 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:21.386752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 629/1250] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:20:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:21.393077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6504d5-64b1-4579-adbd-259d30921633 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:21.489292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:21.518493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6504d5-64b1-4579-adbd-259d30921633 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:21.557907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e571c38-32ad-4107-ada8-dd90f3286815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:21.558581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e571c38-32ad-4107-ada8-dd90f3286815 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:21.558856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e571c38-32ad-4107-ada8-dd90f3286815 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:21.559303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e571c38-32ad-4107-ada8-dd90f3286815 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:21.569249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e571c38-32ad-4107-ada8-dd90f3286815 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:21.570771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e571c38-32ad-4107-ada8-dd90f3286815 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-2e571c38-32ad-4107-ada8-dd90f3286815 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:21.584735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:21.585687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:21.586629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:21.586629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:21.586629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:21.586629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 622/1251] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:21.591875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986e946b-2368-4501-a154-d501ea8373a3 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:21.711170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986e946b-2368-4501-a154-d501ea8373a3 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:21.746486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf51875-6415-411c-8bd5-67c0263348c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:21.747102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf51875-6415-411c-8bd5-67c0263348c1 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:21.747417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf51875-6415-411c-8bd5-67c0263348c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:21.747651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf51875-6415-411c-8bd5-67c0263348c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:21.748285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf51875-6415-411c-8bd5-67c0263348c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:21.748579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf51875-6415-411c-8bd5-67c0263348c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:21.749937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf51875-6415-411c-8bd5-67c0263348c1 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:21.762970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 630/1252] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:21.768879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e571c38-32ad-4107-ada8-dd90f3286815 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:21.877169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e571c38-32ad-4107-ada8-dd90f3286815 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:21.906732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:21.917724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d005c-7af8-4ac9-afbb-f0e3ecf77383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:21.918394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d005c-7af8-4ac9-afbb-f0e3ecf77383 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:21.918635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d005c-7af8-4ac9-afbb-f0e3ecf77383 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:21.918896 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d005c-7af8-4ac9-afbb-f0e3ecf77383 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:21.927050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00d005c-7af8-4ac9-afbb-f0e3ecf77383 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:21.928764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00d005c-7af8-4ac9-afbb-f0e3ecf77383 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-b00d005c-7af8-4ac9-afbb-f0e3ecf77383 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:20:21.940801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:21.942139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:21.943828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 623/1253] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:21 2022] GET /identity/v3/users/placement => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:21.949964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf51875-6415-411c-8bd5-67c0263348c1 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:22.008987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:22.062547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf51875-6415-411c-8bd5-67c0263348c1 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:22.099272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1f674c-e219-4737-a456-b75b1ad3c7d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:22.099885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1f674c-e219-4737-a456-b75b1ad3c7d0 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:22.100157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1f674c-e219-4737-a456-b75b1ad3c7d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:22.100421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1f674c-e219-4737-a456-b75b1ad3c7d0 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:22.101134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1f674c-e219-4737-a456-b75b1ad3c7d0 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:22.101434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1f674c-e219-4737-a456-b75b1ad3c7d0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:22.102886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1f674c-e219-4737-a456-b75b1ad3c7d0 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:22.120623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 631/1254] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Sep 13 15:20:21 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:22.126716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d005c-7af8-4ac9-afbb-f0e3ecf77383 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:22.233582 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d005c-7af8-4ac9-afbb-f0e3ecf77383 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:22.273987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceab9b67-17c9-4aac-88aa-8b0e59c3b021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:22.274899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceab9b67-17c9-4aac-88aa-8b0e59c3b021 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:22.275154 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceab9b67-17c9-4aac-88aa-8b0e59c3b021 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:22.275526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceab9b67-17c9-4aac-88aa-8b0e59c3b021 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:22.333941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:22.353989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceab9b67-17c9-4aac-88aa-8b0e59c3b021 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:22.355627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceab9b67-17c9-4aac-88aa-8b0e59c3b021 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:22.461703 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 624/1255] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:20:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:20:22.467966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1f674c-e219-4737-a456-b75b1ad3c7d0 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:22.574603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1f674c-e219-4737-a456-b75b1ad3c7d0 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:22.587965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:22.610987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105a74c2-3b7a-40d1-ac81-47151a803979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:22.611659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105a74c2-3b7a-40d1-ac81-47151a803979 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:22.611977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105a74c2-3b7a-40d1-ac81-47151a803979 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:22.612281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105a74c2-3b7a-40d1-ac81-47151a803979 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:22.623686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105a74c2-3b7a-40d1-ac81-47151a803979 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:22.625373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105a74c2-3b7a-40d1-ac81-47151a803979 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-105a74c2-3b7a-40d1-ac81-47151a803979 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:22.647650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:22.648920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:22.649907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:22.649907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:22.649907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:22.649907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 632/1256] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:22.654654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceab9b67-17c9-4aac-88aa-8b0e59c3b021 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:22.747409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:22.763115 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceab9b67-17c9-4aac-88aa-8b0e59c3b021 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:22.799531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371df213-e7ae-4592-b3a9-bd8aafc026ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:22.800249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371df213-e7ae-4592-b3a9-bd8aafc026ef None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:22.800508 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371df213-e7ae-4592-b3a9-bd8aafc026ef None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:22.800678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371df213-e7ae-4592-b3a9-bd8aafc026ef None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:22.801357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371df213-e7ae-4592-b3a9-bd8aafc026ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:22.801649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371df213-e7ae-4592-b3a9-bd8aafc026ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:22.803336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371df213-e7ae-4592-b3a9-bd8aafc026ef None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:22.815734 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 625/1257] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:22.821888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105a74c2-3b7a-40d1-ac81-47151a803979 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:22.935939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105a74c2-3b7a-40d1-ac81-47151a803979 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:22.972004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6568187e-aa8e-4d9b-8c07-6fc2bbf2be02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:22.972934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6568187e-aa8e-4d9b-8c07-6fc2bbf2be02 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:22.973574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6568187e-aa8e-4d9b-8c07-6fc2bbf2be02 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:22.973846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6568187e-aa8e-4d9b-8c07-6fc2bbf2be02 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:22.982768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6568187e-aa8e-4d9b-8c07-6fc2bbf2be02 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:22.984288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6568187e-aa8e-4d9b-8c07-6fc2bbf2be02 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-6568187e-aa8e-4d9b-8c07-6fc2bbf2be02 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:20:22.996402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:22.997417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:22.998360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:20:22.998360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:20:22.998360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:20:22.998360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:20:22.998360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:20:22.998360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:20:22.998360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:22.998360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:22.998360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 633/1258] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:20:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:20:23.005556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371df213-e7ae-4592-b3a9-bd8aafc026ef None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:23.120941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371df213-e7ae-4592-b3a9-bd8aafc026ef None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:23.157531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb220c-b8ab-43d8-af8c-497ec8b166e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:23.158203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb220c-b8ab-43d8-af8c-497ec8b166e3 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:23.158483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb220c-b8ab-43d8-af8c-497ec8b166e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:23.158755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb220c-b8ab-43d8-af8c-497ec8b166e3 None admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:23.159459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb220c-b8ab-43d8-af8c-497ec8b166e3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:23.159765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb220c-b8ab-43d8-af8c-497ec8b166e3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:23.161677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb220c-b8ab-43d8-af8c-497ec8b166e3 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:23.175151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 626/1259] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:20:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:23.181806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6568187e-aa8e-4d9b-8c07-6fc2bbf2be02 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:23.210418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:23.292306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6568187e-aa8e-4d9b-8c07-6fc2bbf2be02 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:23.331650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0513fdb8-f05c-4f6a-a113-1199529ef424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:23.332665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0513fdb8-f05c-4f6a-a113-1199529ef424 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:23.333236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0513fdb8-f05c-4f6a-a113-1199529ef424 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:23.333514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0513fdb8-f05c-4f6a-a113-1199529ef424 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:23.334321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0513fdb8-f05c-4f6a-a113-1199529ef424 None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=4bc6be7644b24ca9bab23128b41a7148) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:23.334631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0513fdb8-f05c-4f6a-a113-1199529ef424 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:23.336217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0513fdb8-f05c-4f6a-a113-1199529ef424 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:23.343945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 634/1260] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:20:23 2022] GET /identity/v3/role_assignments?user.id=4bc6be7644b24ca9bab23128b41a7148&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=22819414415942d399db534708fe7103 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:25.489204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94bea84-ac26-4ecb-ac7d-bc647f96bcee None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:25.489771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94bea84-ac26-4ecb-ac7d-bc647f96bcee None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:25.489771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94bea84-ac26-4ecb-ac7d-bc647f96bcee None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:25.491150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 627/1261] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:25.497373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd3af3d-b990-484b-965f-a0a8b7d7380d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:25.498051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd3af3d-b990-484b-965f-a0a8b7d7380d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:25.498193 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd3af3d-b990-484b-965f-a0a8b7d7380d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:25.525776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-3fd3af3d-b990-484b-965f-a0a8b7d7380d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:25.526987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3fd3af3d-b990-484b-965f-a0a8b7d7380d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:25.596357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 635/1262] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:25.861169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94bea84-ac26-4ecb-ac7d-bc647f96bcee None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:25.878060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:25.994292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94bea84-ac26-4ecb-ac7d-bc647f96bcee None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:26.032932 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a222bf60-f1b3-4b20-b8c3-ad71df5b864c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:26.033563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a222bf60-f1b3-4b20-b8c3-ad71df5b864c None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:26.033839 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a222bf60-f1b3-4b20-b8c3-ad71df5b864c None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:26.034150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a222bf60-f1b3-4b20-b8c3-ad71df5b864c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:26.047245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a222bf60-f1b3-4b20-b8c3-ad71df5b864c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-a222bf60-f1b3-4b20-b8c3-ad71df5b864c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:20:26.052366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:26.053398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:26.054409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 628/1263] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:20:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:26.062250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd3af3d-b990-484b-965f-a0a8b7d7380d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:26.082782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:26.172858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd3af3d-b990-484b-965f-a0a8b7d7380d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:26.210246 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6356ad8-f028-4c70-8a53-c3c1a1c0ba82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:26.211056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6356ad8-f028-4c70-8a53-c3c1a1c0ba82 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:26.211330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6356ad8-f028-4c70-8a53-c3c1a1c0ba82 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:26.211582 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6356ad8-f028-4c70-8a53-c3c1a1c0ba82 None admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:26.212240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6356ad8-f028-4c70-8a53-c3c1a1c0ba82 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:26.212536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6356ad8-f028-4c70-8a53-c3c1a1c0ba82 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:26.213903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6356ad8-f028-4c70-8a53-c3c1a1c0ba82 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:26.222699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 636/1264] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:20:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:20:26.229413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a222bf60-f1b3-4b20-b8c3-ad71df5b864c None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:26.309033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:26.356999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a222bf60-f1b3-4b20-b8c3-ad71df5b864c None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:26.403230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d67b9d-2a8b-4a26-8b3b-932e0ebf4369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:26.403873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d67b9d-2a8b-4a26-8b3b-932e0ebf4369 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:26.404141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d67b9d-2a8b-4a26-8b3b-932e0ebf4369 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:26.404420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d67b9d-2a8b-4a26-8b3b-932e0ebf4369 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:26.416819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d67b9d-2a8b-4a26-8b3b-932e0ebf4369 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:26.418928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d67b9d-2a8b-4a26-8b3b-932e0ebf4369 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-96d67b9d-2a8b-4a26-8b3b-932e0ebf4369 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:26.433923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:26.435143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:26.436088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:26.436088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:26.436088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:26.436088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 629/1265] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:26.441416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6356ad8-f028-4c70-8a53-c3c1a1c0ba82 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:26.548687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:26.570616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6356ad8-f028-4c70-8a53-c3c1a1c0ba82 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:26.610217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec333f27-bb25-438b-b77c-3a251b6ea269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:26.610834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec333f27-bb25-438b-b77c-3a251b6ea269 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:26.611111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec333f27-bb25-438b-b77c-3a251b6ea269 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:26.611383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec333f27-bb25-438b-b77c-3a251b6ea269 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:26.612047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec333f27-bb25-438b-b77c-3a251b6ea269 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:26.612343 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec333f27-bb25-438b-b77c-3a251b6ea269 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:26.614066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec333f27-bb25-438b-b77c-3a251b6ea269 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:26.631365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 637/1266] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:26.635120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d67b9d-2a8b-4a26-8b3b-932e0ebf4369 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:26.767803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d67b9d-2a8b-4a26-8b3b-932e0ebf4369 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:26.794806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:26.817790 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d36c6f-de37-4011-99e8-d41acfdeab77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:26.818464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d36c6f-de37-4011-99e8-d41acfdeab77 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:26.818738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d36c6f-de37-4011-99e8-d41acfdeab77 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:26.818995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d36c6f-de37-4011-99e8-d41acfdeab77 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:26.831241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d36c6f-de37-4011-99e8-d41acfdeab77 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:26.833063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d36c6f-de37-4011-99e8-d41acfdeab77 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-90d36c6f-de37-4011-99e8-d41acfdeab77 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:20:26.845244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:26.846400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:26.848060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 630/1267] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:26 2022] GET /identity/v3/users/placement => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:26.853534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec333f27-bb25-438b-b77c-3a251b6ea269 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:26.964587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec333f27-bb25-438b-b77c-3a251b6ea269 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:27.002379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe1753d-e9c3-436e-913f-baaa8f4c2538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:27.002988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe1753d-e9c3-436e-913f-baaa8f4c2538 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:27.003261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe1753d-e9c3-436e-913f-baaa8f4c2538 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:27.003529 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe1753d-e9c3-436e-913f-baaa8f4c2538 None admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:27.004284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe1753d-e9c3-436e-913f-baaa8f4c2538 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:27.004589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe1753d-e9c3-436e-913f-baaa8f4c2538 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:27.006005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe1753d-e9c3-436e-913f-baaa8f4c2538 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:27.015863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:27.025763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 638/1268] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Sep 13 15:20:26 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:20:27.034875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d36c6f-de37-4011-99e8-d41acfdeab77 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:27.143675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d36c6f-de37-4011-99e8-d41acfdeab77 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:27.180122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9236628a-f26e-4faf-9ffe-9638ae5906cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:27.180751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9236628a-f26e-4faf-9ffe-9638ae5906cb None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:27.181038 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9236628a-f26e-4faf-9ffe-9638ae5906cb None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:27.181292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9236628a-f26e-4faf-9ffe-9638ae5906cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:27.192805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9236628a-f26e-4faf-9ffe-9638ae5906cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:27.194280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9236628a-f26e-4faf-9ffe-9638ae5906cb None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-9236628a-f26e-4faf-9ffe-9638ae5906cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:27.208685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:27.211497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:27.211497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:27.211497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:27.211497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 631/1269] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:27.215734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe1753d-e9c3-436e-913f-baaa8f4c2538 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:27.319626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe1753d-e9c3-436e-913f-baaa8f4c2538 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:27.359673 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e4403c-543e-4468-a144-2416c7d0cbb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:27.360291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e4403c-543e-4468-a144-2416c7d0cbb5 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:27.360619 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e4403c-543e-4468-a144-2416c7d0cbb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:27.360874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e4403c-543e-4468-a144-2416c7d0cbb5 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:27.361602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e4403c-543e-4468-a144-2416c7d0cbb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:27.361921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e4403c-543e-4468-a144-2416c7d0cbb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:27.363560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e4403c-543e-4468-a144-2416c7d0cbb5 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:27.375807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 639/1270] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:27.382520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9236628a-f26e-4faf-9ffe-9638ae5906cb None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:27.422979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:27.497608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9236628a-f26e-4faf-9ffe-9638ae5906cb None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:27.533980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b310757-7245-400d-897a-00abe6686dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:27.534631 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b310757-7245-400d-897a-00abe6686dc2 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:27.534940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b310757-7245-400d-897a-00abe6686dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:27.535216 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b310757-7245-400d-897a-00abe6686dc2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:27.542031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b310757-7245-400d-897a-00abe6686dc2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:27.543531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b310757-7245-400d-897a-00abe6686dc2 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-3b310757-7245-400d-897a-00abe6686dc2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:20:27.555345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:27.556319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:27.558245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:20:27.558245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:20:27.558245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:20:27.558245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:20:27.558245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:20:27.558245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:20:27.558245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:27.558245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:27.558245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 632/1271] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:20:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:27.565325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e4403c-543e-4468-a144-2416c7d0cbb5 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:27.626078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:27.670883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e4403c-543e-4468-a144-2416c7d0cbb5 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:27.705511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d3f5d5-a43a-4688-a1f7-f6ba8232557b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:27.706136 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3f5d5-a43a-4688-a1f7-f6ba8232557b None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:27.706436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3f5d5-a43a-4688-a1f7-f6ba8232557b None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:27.706689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3f5d5-a43a-4688-a1f7-f6ba8232557b None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:27.707446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d3f5d5-a43a-4688-a1f7-f6ba8232557b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:27.707770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d3f5d5-a43a-4688-a1f7-f6ba8232557b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:27.709199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d3f5d5-a43a-4688-a1f7-f6ba8232557b None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:27.722605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 640/1272] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:20:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:27.728861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b310757-7245-400d-897a-00abe6686dc2 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:27.836733 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:27.846748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b310757-7245-400d-897a-00abe6686dc2 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:27.882560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25539168-d85e-4da0-ba85-0d9d643791b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:27.883143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25539168-d85e-4da0-ba85-0d9d643791b7 None admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:27.883415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25539168-d85e-4da0-ba85-0d9d643791b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:27.883683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25539168-d85e-4da0-ba85-0d9d643791b7 None admin] PATH_INFO: `/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/4bc6be7644b24ca9bab23128b41a7148/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:27.885623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25539168-d85e-4da0-ba85-0d9d643791b7 None admin] RBAC: Authorizing `identity:create_grant(project_id=c30b8f78ae8c4ee2b59357d0be54f953, user_id=4bc6be7644b24ca9bab23128b41a7148, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:27.887162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25539168-d85e-4da0-ba85-0d9d643791b7 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:27.907062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 633/1273] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Sep 13 15:20:27 2022] PUT /identity/v3/projects/c30b8f78ae8c4ee2b59357d0be54f953/users/4bc6be7644b24ca9bab23128b41a7148/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:20:30.079818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355e6026-afaf-4f5c-92b5-da72d677ca0a None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:30.080372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355e6026-afaf-4f5c-92b5-da72d677ca0a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:30.080372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355e6026-afaf-4f5c-92b5-da72d677ca0a None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:30.082363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 641/1274] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:30.087873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726b0203-a1be-4124-a2ed-9347183937f3 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:30.088293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726b0203-a1be-4124-a2ed-9347183937f3 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:30.088410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726b0203-a1be-4124-a2ed-9347183937f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:30.109440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-726b0203-a1be-4124-a2ed-9347183937f3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:30.110672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-726b0203-a1be-4124-a2ed-9347183937f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:30.191438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 634/1275] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:30.470384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355e6026-afaf-4f5c-92b5-da72d677ca0a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:30.601788 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355e6026-afaf-4f5c-92b5-da72d677ca0a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:30.620118 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:30.641864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255623e0-42f0-4573-bab1-d56033fd8185 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:30.642535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255623e0-42f0-4573-bab1-d56033fd8185 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:30.642810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255623e0-42f0-4573-bab1-d56033fd8185 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:30.643082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255623e0-42f0-4573-bab1-d56033fd8185 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:30.653813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255623e0-42f0-4573-bab1-d56033fd8185 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-255623e0-42f0-4573-bab1-d56033fd8185 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 13 15:20:30.658645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:30.659817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:30.661048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 642/1276] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Sep 13 15:20:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:30.666037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726b0203-a1be-4124-a2ed-9347183937f3 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:30.748872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:30.775676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726b0203-a1be-4124-a2ed-9347183937f3 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:30.808489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fcc886-8473-48fa-87d6-977a91591906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:30.809090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fcc886-8473-48fa-87d6-977a91591906 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:30.809361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fcc886-8473-48fa-87d6-977a91591906 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:30.809629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fcc886-8473-48fa-87d6-977a91591906 None admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:30.810321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fcc886-8473-48fa-87d6-977a91591906 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:30.810621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fcc886-8473-48fa-87d6-977a91591906 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:30.811992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fcc886-8473-48fa-87d6-977a91591906 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:30.820131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 635/1277] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Sep 13 15:20:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:20:30.827409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255623e0-42f0-4573-bab1-d56033fd8185 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:30.940231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255623e0-42f0-4573-bab1-d56033fd8185 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:30.975406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa20953-13dd-4214-bbdd-95c4f33a94c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:30.976046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa20953-13dd-4214-bbdd-95c4f33a94c8 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:30.976311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa20953-13dd-4214-bbdd-95c4f33a94c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:30.976582 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa20953-13dd-4214-bbdd-95c4f33a94c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:30.987359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa20953-13dd-4214-bbdd-95c4f33a94c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:30.988814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa20953-13dd-4214-bbdd-95c4f33a94c8 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-4aa20953-13dd-4214-bbdd-95c4f33a94c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:31.004043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:31.009986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:31.011177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:31.011177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:31.011177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:31.011177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 643/1278] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:31.012752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fcc886-8473-48fa-87d6-977a91591906 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:31.117074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fcc886-8473-48fa-87d6-977a91591906 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:31.151592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc832840-844d-41be-93d4-b7a849130dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:31.152188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc832840-844d-41be-93d4-b7a849130dfd None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:31.152458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc832840-844d-41be-93d4-b7a849130dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:31.152725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc832840-844d-41be-93d4-b7a849130dfd None admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:31.153352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc832840-844d-41be-93d4-b7a849130dfd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:31.153649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc832840-844d-41be-93d4-b7a849130dfd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:31.155053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc832840-844d-41be-93d4-b7a849130dfd None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:31.166722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 636/1279] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:31.175145 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa20953-13dd-4214-bbdd-95c4f33a94c8 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:31.206622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:31.276676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa20953-13dd-4214-bbdd-95c4f33a94c8 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:31.311288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a3f4b0-0eb3-4bbc-85ff-6f70e913b306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:31.312263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a3f4b0-0eb3-4bbc-85ff-6f70e913b306 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:31.312689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a3f4b0-0eb3-4bbc-85ff-6f70e913b306 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:31.313092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a3f4b0-0eb3-4bbc-85ff-6f70e913b306 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:31.326166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a3f4b0-0eb3-4bbc-85ff-6f70e913b306 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:31.328598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a3f4b0-0eb3-4bbc-85ff-6f70e913b306 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-a6a3f4b0-0eb3-4bbc-85ff-6f70e913b306 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 13 15:20:31.342661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:31.343699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:31.344826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 644/1280] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:31 2022] GET /identity/v3/users/placement => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:20:31.350808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc832840-844d-41be-93d4-b7a849130dfd None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:31.354758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:31.459825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc832840-844d-41be-93d4-b7a849130dfd None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:31.495359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed2e02-567a-4d22-b817-148ea93ed39e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:31.495955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed2e02-567a-4d22-b817-148ea93ed39e None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:31.496229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed2e02-567a-4d22-b817-148ea93ed39e None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:31.496500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed2e02-567a-4d22-b817-148ea93ed39e None admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:31.497221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faed2e02-567a-4d22-b817-148ea93ed39e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:31.497528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faed2e02-567a-4d22-b817-148ea93ed39e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:31.498986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faed2e02-567a-4d22-b817-148ea93ed39e None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:31.516141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 637/1281] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Sep 13 15:20:31 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:31.522409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a3f4b0-0eb3-4bbc-85ff-6f70e913b306 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:31.620873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:31.633505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a3f4b0-0eb3-4bbc-85ff-6f70e913b306 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:31.668305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e11c7a9-e37f-40e2-b87c-dd7201837a37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:31.668909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e11c7a9-e37f-40e2-b87c-dd7201837a37 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:31.669180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e11c7a9-e37f-40e2-b87c-dd7201837a37 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:31.669448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e11c7a9-e37f-40e2-b87c-dd7201837a37 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:31.737704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e11c7a9-e37f-40e2-b87c-dd7201837a37 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:31.739163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e11c7a9-e37f-40e2-b87c-dd7201837a37 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:31.827352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:31.838581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 645/1282] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Sep 13 15:20:31 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 13 15:20:31.845451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed2e02-567a-4d22-b817-148ea93ed39e None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:31.910599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:31.949629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed2e02-567a-4d22-b817-148ea93ed39e None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:31.993456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124f16b-ffcc-4b82-9c16-afb0187e1233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:31.994488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124f16b-ffcc-4b82-9c16-afb0187e1233 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:31.994952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124f16b-ffcc-4b82-9c16-afb0187e1233 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:31.995393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124f16b-ffcc-4b82-9c16-afb0187e1233 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:32.009718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c124f16b-ffcc-4b82-9c16-afb0187e1233 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:32.011212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c124f16b-ffcc-4b82-9c16-afb0187e1233 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-c124f16b-ffcc-4b82-9c16-afb0187e1233 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 13 15:20:32.025785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:32.026852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 13 15:20:32.027758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 13 15:20:32.027758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 13 15:20:32.027758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:32.027758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 638/1283] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:20:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:32.035512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e11c7a9-e37f-40e2-b87c-dd7201837a37 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:32.150579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e11c7a9-e37f-40e2-b87c-dd7201837a37 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:32.188423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd469099-963e-42d4-8ec5-a534dd595e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:32.189022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd469099-963e-42d4-8ec5-a534dd595e27 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:32.189195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd469099-963e-42d4-8ec5-a534dd595e27 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:32.189468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd469099-963e-42d4-8ec5-a534dd595e27 None admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:32.190132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd469099-963e-42d4-8ec5-a534dd595e27 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:32.190438 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd469099-963e-42d4-8ec5-a534dd595e27 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:32.191849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd469099-963e-42d4-8ec5-a534dd595e27 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:32.203021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 646/1284] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Sep 13 15:20:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:32.209254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124f16b-ffcc-4b82-9c16-afb0187e1233 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:32.313798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124f16b-ffcc-4b82-9c16-afb0187e1233 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:32.327944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:32.351125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f290a2fa-ca0f-401a-a8ac-29db1f120095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:32.351754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f290a2fa-ca0f-401a-a8ac-29db1f120095 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:32.352034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f290a2fa-ca0f-401a-a8ac-29db1f120095 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:32.352305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f290a2fa-ca0f-401a-a8ac-29db1f120095 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:32.360179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f290a2fa-ca0f-401a-a8ac-29db1f120095 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:32.361636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f290a2fa-ca0f-401a-a8ac-29db1f120095 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-f290a2fa-ca0f-401a-a8ac-29db1f120095 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 13 15:20:32.373727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:32.374796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:32.376017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:20:32.376017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:20:32.376017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:20:32.376017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:20:32.376017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:20:32.376017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:20:32.376017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 13 15:20:32.376017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:20:32.376017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 639/1285] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Sep 13 15:20:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:32.382580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd469099-963e-42d4-8ec5-a534dd595e27 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:32.433379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:32.485435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd469099-963e-42d4-8ec5-a534dd595e27 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:32.528183 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834ce06a-c334-41a9-881e-9ab97cd24158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:32.529106 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834ce06a-c334-41a9-881e-9ab97cd24158 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:32.529523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834ce06a-c334-41a9-881e-9ab97cd24158 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:32.529938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834ce06a-c334-41a9-881e-9ab97cd24158 None admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:32.531047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834ce06a-c334-41a9-881e-9ab97cd24158 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:32.531517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834ce06a-c334-41a9-881e-9ab97cd24158 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:32.533771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834ce06a-c334-41a9-881e-9ab97cd24158 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:32.546314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 647/1286] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Sep 13 15:20:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:32.552652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f290a2fa-ca0f-401a-a8ac-29db1f120095 None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:32.683225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f290a2fa-ca0f-401a-a8ac-29db1f120095 None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:32.718298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1939d58a-a561-45fc-bb7d-698f5cef830a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:32.718897 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1939d58a-a561-45fc-bb7d-698f5cef830a None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:32.719164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1939d58a-a561-45fc-bb7d-698f5cef830a None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:32.719430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1939d58a-a561-45fc-bb7d-698f5cef830a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:32.720145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1939d58a-a561-45fc-bb7d-698f5cef830a None admin] RBAC: Adding query filter params (role.id=22819414415942d399db534708fe7103, scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953, user.id=4bc6be7644b24ca9bab23128b41a7148) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:32.720443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1939d58a-a561-45fc-bb7d-698f5cef830a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:32.721929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1939d58a-a561-45fc-bb7d-698f5cef830a None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:32.727791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 640/1287] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Sep 13 15:20:32 2022] GET /identity/v3/role_assignments?user.id=4bc6be7644b24ca9bab23128b41a7148&scope.project.id=c30b8f78ae8c4ee2b59357d0be54f953&role.id=22819414415942d399db534708fe7103 => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:34.871693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cec7784-3c45-46eb-822c-9f73635e1893 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:34.872580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cec7784-3c45-46eb-822c-9f73635e1893 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:34.872580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cec7784-3c45-46eb-822c-9f73635e1893 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:34.874746 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 648/1288] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:34.880235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd13fd-cd2b-414d-bc89-dc804af007dc None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:34.880683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd13fd-cd2b-414d-bc89-dc804af007dc None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:34.880775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd13fd-cd2b-414d-bc89-dc804af007dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:34.900674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-55bd13fd-cd2b-414d-bc89-dc804af007dc None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:34.901857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-55bd13fd-cd2b-414d-bc89-dc804af007dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:34.936634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:34.987069 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 641/1289] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:35.263383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cec7784-3c45-46eb-822c-9f73635e1893 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:35.381559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cec7784-3c45-46eb-822c-9f73635e1893 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:35.395270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:35.420389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee04dee-389e-4c32-8b1e-d77e0c797123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:35.421021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee04dee-389e-4c32-8b1e-d77e0c797123 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:35.421294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee04dee-389e-4c32-8b1e-d77e0c797123 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:35.421562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee04dee-389e-4c32-8b1e-d77e0c797123 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:35.422163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee04dee-389e-4c32-8b1e-d77e0c797123 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:35.423524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee04dee-389e-4c32-8b1e-d77e0c797123 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-8ee04dee-389e-4c32-8b1e-d77e0c797123 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:35.432637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:35.433798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:35.434708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 649/1290] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Sep 13 15:20:35 2022] GET /identity/v3/services/placement => generated 90 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:35.440133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd13fd-cd2b-414d-bc89-dc804af007dc None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:35.547137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd13fd-cd2b-414d-bc89-dc804af007dc None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:35.586611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f69087f-2506-4161-a514-a03b35e157ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:35.587467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69087f-2506-4161-a514-a03b35e157ae None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:35.587749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69087f-2506-4161-a514-a03b35e157ae None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:35.588018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69087f-2506-4161-a514-a03b35e157ae None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:35.588653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f69087f-2506-4161-a514-a03b35e157ae None admin] RBAC: Adding query filter params (name=placement) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:35.588953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f69087f-2506-4161-a514-a03b35e157ae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:35.590351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f69087f-2506-4161-a514-a03b35e157ae None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:35.595574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 642/1291] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Sep 13 15:20:35 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:35.605492 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee04dee-389e-4c32-8b1e-d77e0c797123 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:35.726004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee04dee-389e-4c32-8b1e-d77e0c797123 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:35.761548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153371bf-471f-4f37-a928-145c194477a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:35.762136 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153371bf-471f-4f37-a928-145c194477a0 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:35.762448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153371bf-471f-4f37-a928-145c194477a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:35.762722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153371bf-471f-4f37-a928-145c194477a0 None admin] PATH_INFO: `/v3/services` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:35.763346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153371bf-471f-4f37-a928-145c194477a0 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:35.763653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153371bf-471f-4f37-a928-145c194477a0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:35.765024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153371bf-471f-4f37-a928-145c194477a0 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:35.769601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 650/1292] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Sep 13 15:20:35 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:37.943078 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7fc03a-c14e-4cdc-9935-cce34aa7d3de None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:37.943963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7fc03a-c14e-4cdc-9935-cce34aa7d3de None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:37.943963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7fc03a-c14e-4cdc-9935-cce34aa7d3de None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:37.946178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 643/1293] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:37.954136 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff21be0-167a-4338-b2b4-ba297bc8485e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:37.954593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff21be0-167a-4338-b2b4-ba297bc8485e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:37.954696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff21be0-167a-4338-b2b4-ba297bc8485e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:37.975859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-aff21be0-167a-4338-b2b4-ba297bc8485e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:37.977038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-aff21be0-167a-4338-b2b4-ba297bc8485e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:38.008360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:38.047986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 651/1294] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:38.320835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7fc03a-c14e-4cdc-9935-cce34aa7d3de None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:38.365353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:38.457389 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7fc03a-c14e-4cdc-9935-cce34aa7d3de None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:38.507159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6d834c-f8ae-4991-bab6-06a31c622aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:38.507764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6d834c-f8ae-4991-bab6-06a31c622aa9 None admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:38.508039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6d834c-f8ae-4991-bab6-06a31c622aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:38.508305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6d834c-f8ae-4991-bab6-06a31c622aa9 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:38.508952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6d834c-f8ae-4991-bab6-06a31c622aa9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:38.510644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6d834c-f8ae-4991-bab6-06a31c622aa9 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:38.528918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 644/1295] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Sep 13 15:20:38 2022] POST /identity/v3/services => generated 246 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:20:40.674148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c08d28-2181-44c4-9354-e1656a8a27e7 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:40.674983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c08d28-2181-44c4-9354-e1656a8a27e7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:40.675103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c08d28-2181-44c4-9354-e1656a8a27e7 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:40.677731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 652/1296] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:40.684288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255e7893-3abb-484b-b88f-9a7c50164340 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:40.685005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255e7893-3abb-484b-b88f-9a7c50164340 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:40.685167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255e7893-3abb-484b-b88f-9a7c50164340 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:40.707282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-255e7893-3abb-484b-b88f-9a7c50164340 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:40.708459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-255e7893-3abb-484b-b88f-9a7c50164340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:40.795544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 645/1297] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:40 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:41.069677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c08d28-2181-44c4-9354-e1656a8a27e7 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:41.180496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c08d28-2181-44c4-9354-e1656a8a27e7 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:41.215046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe687fa-5001-4208-8384-8dae1b68b2d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:41.215655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe687fa-5001-4208-8384-8dae1b68b2d9 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:41.215929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe687fa-5001-4208-8384-8dae1b68b2d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:41.216194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe687fa-5001-4208-8384-8dae1b68b2d9 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:41.216769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe687fa-5001-4208-8384-8dae1b68b2d9 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:41.218146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe687fa-5001-4208-8384-8dae1b68b2d9 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-9fe687fa-5001-4208-8384-8dae1b68b2d9 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:41.227069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:41.228174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:41.229326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 653/1298] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Sep 13 15:20:41 2022] GET /identity/v3/services/placement => generated 90 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 13 15:20:41.237913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255e7893-3abb-484b-b88f-9a7c50164340 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:41.262567 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:41.352198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255e7893-3abb-484b-b88f-9a7c50164340 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:41.391245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ba678b-f0d5-43fb-8e73-8c442665b8cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:41.391865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba678b-f0d5-43fb-8e73-8c442665b8cf None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:41.392143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba678b-f0d5-43fb-8e73-8c442665b8cf None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:41.392415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba678b-f0d5-43fb-8e73-8c442665b8cf None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:41.393151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ba678b-f0d5-43fb-8e73-8c442665b8cf None admin] RBAC: Adding query filter params (name=placement) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:41.393465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ba678b-f0d5-43fb-8e73-8c442665b8cf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:41.395340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ba678b-f0d5-43fb-8e73-8c442665b8cf None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:41.400489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 646/1299] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Sep 13 15:20:41 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:41.409769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe687fa-5001-4208-8384-8dae1b68b2d9 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:41.444921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:41.515837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe687fa-5001-4208-8384-8dae1b68b2d9 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:41.550710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8248c4-66f6-4dda-bac9-6d8d0a930887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:41.551300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8248c4-66f6-4dda-bac9-6d8d0a930887 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:41.551575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8248c4-66f6-4dda-bac9-6d8d0a930887 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:41.551844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8248c4-66f6-4dda-bac9-6d8d0a930887 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:41.552532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8248c4-66f6-4dda-bac9-6d8d0a930887 None admin] RBAC: Adding query filter params (interface=public, service_id=77be2162b8e2442988da0e02f3ad5352, region_id=RegionOne) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:41.552843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8248c4-66f6-4dda-bac9-6d8d0a930887 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:41.554224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8248c4-66f6-4dda-bac9-6d8d0a930887 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:41.559292 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 654/1300] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Sep 13 15:20:41 2022] GET /identity/v3/endpoints?service_id=77be2162b8e2442988da0e02f3ad5352&interface=public®ion_id=RegionOne => generated 194 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:41.566633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ba678b-f0d5-43fb-8e73-8c442665b8cf None admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:41.665242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:41.691752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ba678b-f0d5-43fb-8e73-8c442665b8cf None admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:41.728359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeab1bea-741b-4c7b-bb74-795ed50dbf66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:41.728960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeab1bea-741b-4c7b-bb74-795ed50dbf66 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:41.729236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeab1bea-741b-4c7b-bb74-795ed50dbf66 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:41.729506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeab1bea-741b-4c7b-bb74-795ed50dbf66 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:41.730483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeab1bea-741b-4c7b-bb74-795ed50dbf66 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:41.731857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeab1bea-741b-4c7b-bb74-795ed50dbf66 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:41.738884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 647/1301] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Sep 13 15:20:41 2022] GET /identity/v3/services => generated 2191 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:20:43.850312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a0b95-b567-4034-a9c2-602ae05caea0 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:43.850955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a0b95-b567-4034-a9c2-602ae05caea0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:43.850955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a0b95-b567-4034-a9c2-602ae05caea0 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:43.852194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 655/1302] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:20:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:20:43.858919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b10115-9e35-4727-bc81-b674ba6380c5 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:43.859648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b10115-9e35-4727-bc81-b674ba6380c5 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:43.859794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b10115-9e35-4727-bc81-b674ba6380c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:43.886052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e8b10115-9e35-4727-bc81-b674ba6380c5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:20:43.887305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e8b10115-9e35-4727-bc81-b674ba6380c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:20:43.955250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 648/1303] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:20:43 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 13 15:20:44.223391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9a0b95-b567-4034-a9c2-602ae05caea0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:44.325179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:44.333218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9a0b95-b567-4034-a9c2-602ae05caea0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:44.369387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef4bd8a-0542-4f9a-a473-a11544a5c3b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:44.370009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef4bd8a-0542-4f9a-a473-a11544a5c3b4 None admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:44.370315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef4bd8a-0542-4f9a-a473-a11544a5c3b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:44.370578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef4bd8a-0542-4f9a-a473-a11544a5c3b4 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:44.371156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef4bd8a-0542-4f9a-a473-a11544a5c3b4 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:44.372523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef4bd8a-0542-4f9a-a473-a11544a5c3b4 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application [None req-3ef4bd8a-0542-4f9a-a473-a11544a5c3b4 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:20:44.381509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:20:44.382713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application with Lock( Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: ERROR keystone.server.flask.application Sep 13 15:20:44.383628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 656/1304] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Sep 13 15:20:44 2022] GET /identity/v3/services/placement => generated 90 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 13 15:20:44.389590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b10115-9e35-4727-bc81-b674ba6380c5 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:44.469044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:44.494259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b10115-9e35-4727-bc81-b674ba6380c5 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:44.530803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2009cf-56cb-4056-9485-5628bb9ab7a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:44.531390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2009cf-56cb-4056-9485-5628bb9ab7a7 None admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:44.531661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2009cf-56cb-4056-9485-5628bb9ab7a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:44.531913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2009cf-56cb-4056-9485-5628bb9ab7a7 None admin] PATH_INFO: `/v3/services` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:44.532533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2009cf-56cb-4056-9485-5628bb9ab7a7 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:20:44.532823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2009cf-56cb-4056-9485-5628bb9ab7a7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:44.534293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2009cf-56cb-4056-9485-5628bb9ab7a7 None admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:44.539756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 649/1305] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Sep 13 15:20:44 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:20:44.548476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef4bd8a-0542-4f9a-a473-a11544a5c3b4 None admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:20:44.654727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef4bd8a-0542-4f9a-a473-a11544a5c3b4 None admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:20:44.691314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ba8dbc-a69a-4579-92f6-abea3bf47705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:20:44.691909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ba8dbc-a69a-4579-92f6-abea3bf47705 None admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:20:44.692186 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ba8dbc-a69a-4579-92f6-abea3bf47705 None admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:20:44.692451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ba8dbc-a69a-4579-92f6-abea3bf47705 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:20:44.693127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ba8dbc-a69a-4579-92f6-abea3bf47705 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:20:44.694516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ba8dbc-a69a-4579-92f6-abea3bf47705 None admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:20:44.720616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 657/1306] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:20:44 2022] POST /identity/v3/endpoints => generated 331 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:20:54.536822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=30 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:20:54.714699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:21:49.421961 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc8ecb8-412e-4033-b30e-13ded4f2d999 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:49.426455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc8ecb8-412e-4033-b30e-13ded4f2d999 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:49.426455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc8ecb8-412e-4033-b30e-13ded4f2d999 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:49.428150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 650/1307] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:21:49 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:21:49.429550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9b5813-b743-46e8-bea0-a5344fa624e5 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:49.430613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9b5813-b743-46e8-bea0-a5344fa624e5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:49.431137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9b5813-b743-46e8-bea0-a5344fa624e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:49.565781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-ca9b5813-b743-46e8-bea0-a5344fa624e5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:21:49.567347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ca9b5813-b743-46e8-bea0-a5344fa624e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:21:49.687496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 658/1308] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:21:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:21:49.978897 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91337325-7e3b-4975-b06a-35416a9597db None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:49.979180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91337325-7e3b-4975-b06a-35416a9597db None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:49.979703 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91337325-7e3b-4975-b06a-35416a9597db None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:49.981113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 651/1309] 10.222.0.16 () {54 vars in 965 bytes} [Tue Sep 13 15:21:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:21:49.987768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e87ee-56a5-4eee-94c6-ba6761e1ff66 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:49.988128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e87ee-56a5-4eee-94c6-ba6761e1ff66 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:49.988474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e87ee-56a5-4eee-94c6-ba6761e1ff66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:50.070899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-5f3e87ee-56a5-4eee-94c6-ba6761e1ff66 None None] MFA Rules not processed for user `171e916244804add8f18b44b77b03d72`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:21:50.072458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5f3e87ee-56a5-4eee-94c6-ba6761e1ff66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:21:50.097656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:21:50.162625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 659/1310] 10.222.0.16 () {58 vars in 1062 bytes} [Tue Sep 13 15:21:49 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:21:50.172517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91337325-7e3b-4975-b06a-35416a9597db None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:21:50.174447 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-91337325-7e3b-4975-b06a-35416a9597db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:21:50.298736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91337325-7e3b-4975-b06a-35416a9597db None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:21:50.381343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f71f70-5ddc-4fce-867a-d16b56fdb183 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:21:50.382077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f71f70-5ddc-4fce-867a-d16b56fdb183 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:50.382521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f71f70-5ddc-4fce-867a-d16b56fdb183 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:50.382792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f71f70-5ddc-4fce-867a-d16b56fdb183 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:50.384540 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-61f71f70-5ddc-4fce-867a-d16b56fdb183 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:21:50.443904 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:21:50.508347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f71f70-5ddc-4fce-867a-d16b56fdb183 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:21:50.509684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f71f70-5ddc-4fce-867a-d16b56fdb183 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:21:50.529609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 652/1311] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:21:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:21:53.018742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b6aa6-15ec-48de-a977-d3ce14ba5cb6 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:53.019331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b6aa6-15ec-48de-a977-d3ce14ba5cb6 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:53.019331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b6aa6-15ec-48de-a977-d3ce14ba5cb6 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:53.021304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 660/1312] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:21:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:21:53.030263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3070c4-52ff-4f2e-883e-90a36150475a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:53.030711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3070c4-52ff-4f2e-883e-90a36150475a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:53.030802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3070c4-52ff-4f2e-883e-90a36150475a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:53.108731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7f3070c4-52ff-4f2e-883e-90a36150475a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:21:53.109853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7f3070c4-52ff-4f2e-883e-90a36150475a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:21:53.210434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 653/1313] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:21:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:21:53.503529 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7b6aa6-15ec-48de-a977-d3ce14ba5cb6 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:21:53.506086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2e7b6aa6-15ec-48de-a977-d3ce14ba5cb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:21:53.639178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7b6aa6-15ec-48de-a977-d3ce14ba5cb6 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:21:53.715507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:21:53.718657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c832f6f-8536-4ed2-9cab-9a503da86e8e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:21:53.719261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c832f6f-8536-4ed2-9cab-9a503da86e8e service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:53.719541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c832f6f-8536-4ed2-9cab-9a503da86e8e service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:53.719815 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c832f6f-8536-4ed2-9cab-9a503da86e8e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:53.721817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9c832f6f-8536-4ed2-9cab-9a503da86e8e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:21:53.833534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c832f6f-8536-4ed2-9cab-9a503da86e8e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:21:53.834948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c832f6f-8536-4ed2-9cab-9a503da86e8e service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:21:53.857601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 661/1314] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:21:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:21:56.315474 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23f2292-1a3a-4347-be2a-d58d6d4e4e7d None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:56.316072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23f2292-1a3a-4347-be2a-d58d6d4e4e7d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:56.316072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23f2292-1a3a-4347-be2a-d58d6d4e4e7d None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:56.318147 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 654/1315] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:21:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:21:56.323570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf7fd4d-a6b0-48a5-a8bb-9fc0212682ce None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:56.324036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf7fd4d-a6b0-48a5-a8bb-9fc0212682ce None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:56.324159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf7fd4d-a6b0-48a5-a8bb-9fc0212682ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:56.396860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0cf7fd4d-a6b0-48a5-a8bb-9fc0212682ce None None] MFA Rules not processed for user `a734913c72ea45f3abe03c5a9b7f9695`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:21:56.398132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0cf7fd4d-a6b0-48a5-a8bb-9fc0212682ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:21:56.451933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:21:56.500885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 662/1316] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:21:56 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:21:56.781997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23f2292-1a3a-4347-be2a-d58d6d4e4e7d None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:21:56.784572 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c23f2292-1a3a-4347-be2a-d58d6d4e4e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:21:56.799741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:21:56.898925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23f2292-1a3a-4347-be2a-d58d6d4e4e7d None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:21:56.980503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb34b1a-e020-4277-93f3-206c2f0dd5e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:21:56.981116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb34b1a-e020-4277-93f3-206c2f0dd5e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:56.981390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb34b1a-e020-4277-93f3-206c2f0dd5e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:56.981819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb34b1a-e020-4277-93f3-206c2f0dd5e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:56.983445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-cfb34b1a-e020-4277-93f3-206c2f0dd5e1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:21:57.047180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:21:57.098790 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb34b1a-e020-4277-93f3-206c2f0dd5e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:21:57.100562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb34b1a-e020-4277-93f3-206c2f0dd5e1 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:21:57.119475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 655/1317] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:21:56 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:21:59.844072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe479036-b5a2-4c67-816e-e12e29f1b0ad None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:59.844656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe479036-b5a2-4c67-816e-e12e29f1b0ad None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:59.844656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe479036-b5a2-4c67-816e-e12e29f1b0ad None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:59.846774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 663/1318] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:21:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:21:59.855155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f42ce2c-4aa4-43f2-941a-9d65813c9854 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:21:59.855600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f42ce2c-4aa4-43f2-941a-9d65813c9854 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:21:59.855692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f42ce2c-4aa4-43f2-941a-9d65813c9854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:21:59.934383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7f42ce2c-4aa4-43f2-941a-9d65813c9854 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:21:59.935957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7f42ce2c-4aa4-43f2-941a-9d65813c9854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:00.039078 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 656/1319] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:21:59 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:22:00.328098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe479036-b5a2-4c67-816e-e12e29f1b0ad None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:00.330649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-fe479036-b5a2-4c67-816e-e12e29f1b0ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:00.457989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe479036-b5a2-4c67-816e-e12e29f1b0ad None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:00.533632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fd275a-e3dc-48f3-9eb1-990a7cc50b97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:00.534258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fd275a-e3dc-48f3-9eb1-990a7cc50b97 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:00.534533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fd275a-e3dc-48f3-9eb1-990a7cc50b97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:00.534808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fd275a-e3dc-48f3-9eb1-990a7cc50b97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:00.536404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e2fd275a-e3dc-48f3-9eb1-990a7cc50b97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:00.543587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:00.668967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fd275a-e3dc-48f3-9eb1-990a7cc50b97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:00.670440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fd275a-e3dc-48f3-9eb1-990a7cc50b97 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:00.688869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 664/1320] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:00 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:03.492665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea40a5e-afa6-4091-85bf-b9e9a47dff48 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:03.493272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea40a5e-afa6-4091-85bf-b9e9a47dff48 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:03.493272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea40a5e-afa6-4091-85bf-b9e9a47dff48 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:03.494595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 657/1321] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:03.500129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1960a35-5cd3-4d16-8a01-5f5c551a83af None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:03.500574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1960a35-5cd3-4d16-8a01-5f5c551a83af None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:03.500653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1960a35-5cd3-4d16-8a01-5f5c551a83af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:03.572357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c1960a35-5cd3-4d16-8a01-5f5c551a83af None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:03.573613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c1960a35-5cd3-4d16-8a01-5f5c551a83af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:03.645977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:03.689107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 665/1322] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:03.966391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea40a5e-afa6-4091-85bf-b9e9a47dff48 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:03.968054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0ea40a5e-afa6-4091-85bf-b9e9a47dff48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:03.989890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:04.078213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea40a5e-afa6-4091-85bf-b9e9a47dff48 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:04.155745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1132ca-d687-4dac-9d3d-908908c05c73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:04.156360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1132ca-d687-4dac-9d3d-908908c05c73 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:04.156636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1132ca-d687-4dac-9d3d-908908c05c73 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:04.156905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1132ca-d687-4dac-9d3d-908908c05c73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:04.158546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6c1132ca-d687-4dac-9d3d-908908c05c73 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:04.235484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:04.274050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1132ca-d687-4dac-9d3d-908908c05c73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:04.275559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1132ca-d687-4dac-9d3d-908908c05c73 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:04.295400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 658/1323] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:06.521063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c055b27-fde2-466a-872e-8ef51f47f9db None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:06.521663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c055b27-fde2-466a-872e-8ef51f47f9db None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:06.521663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c055b27-fde2-466a-872e-8ef51f47f9db None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:06.522967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 666/1324] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:06.528492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8311b60e-8405-4f27-86bc-3823e142020c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:06.528941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8311b60e-8405-4f27-86bc-3823e142020c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:06.529128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8311b60e-8405-4f27-86bc-3823e142020c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:06.598191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8311b60e-8405-4f27-86bc-3823e142020c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:06.599456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8311b60e-8405-4f27-86bc-3823e142020c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:06.719094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 659/1325] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:07.018140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794473e5-bf19-40d1-a477-8fb1daab43ed None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:07.018444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794473e5-bf19-40d1-a477-8fb1daab43ed None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:07.018709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794473e5-bf19-40d1-a477-8fb1daab43ed None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:07.020045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 667/1326] 10.222.0.16 () {54 vars in 965 bytes} [Tue Sep 13 15:22:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:07.025953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf21b8-f088-482b-bee5-5fd8d7ed2962 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:07.026278 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf21b8-f088-482b-bee5-5fd8d7ed2962 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:07.026549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf21b8-f088-482b-bee5-5fd8d7ed2962 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:07.058923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:07.107110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-96bf21b8-f088-482b-bee5-5fd8d7ed2962 None None] MFA Rules not processed for user `171e916244804add8f18b44b77b03d72`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:07.108331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-96bf21b8-f088-482b-bee5-5fd8d7ed2962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:07.187765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 660/1327] 10.222.0.16 () {58 vars in 1062 bytes} [Tue Sep 13 15:22:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:07.197110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794473e5-bf19-40d1-a477-8fb1daab43ed None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:07.198695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-794473e5-bf19-40d1-a477-8fb1daab43ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:07.310613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794473e5-bf19-40d1-a477-8fb1daab43ed None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:07.387700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6654a2-481b-421c-b140-18d0935fdc4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:07.388307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6654a2-481b-421c-b140-18d0935fdc4e service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:07.388577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6654a2-481b-421c-b140-18d0935fdc4e service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:07.388811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6654a2-481b-421c-b140-18d0935fdc4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:07.390457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9b6654a2-481b-421c-b140-18d0935fdc4e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:07.408252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:07.509243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6654a2-481b-421c-b140-18d0935fdc4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:07.510734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6654a2-481b-421c-b140-18d0935fdc4e service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:07.530307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 668/1328] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:10.030840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911522c7-bcc3-4f5c-8f7b-9444964c500e None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:10.031396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911522c7-bcc3-4f5c-8f7b-9444964c500e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:10.031396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911522c7-bcc3-4f5c-8f7b-9444964c500e None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:10.032922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 661/1329] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:10.038545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0981fbb-fd14-459f-9a50-ff499e6b2788 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:10.038998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0981fbb-fd14-459f-9a50-ff499e6b2788 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:10.039081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0981fbb-fd14-459f-9a50-ff499e6b2788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:10.110390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a0981fbb-fd14-459f-9a50-ff499e6b2788 None None] MFA Rules not processed for user `a734913c72ea45f3abe03c5a9b7f9695`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:10.111601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a0981fbb-fd14-459f-9a50-ff499e6b2788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:10.177602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:10.212739 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 669/1330] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:10 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:10.499687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911522c7-bcc3-4f5c-8f7b-9444964c500e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:10.501656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-911522c7-bcc3-4f5c-8f7b-9444964c500e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:10.613217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911522c7-bcc3-4f5c-8f7b-9444964c500e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:10.635125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:10.693352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45566764-9230-4c02-8daa-11a1bba70861 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:10.693950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45566764-9230-4c02-8daa-11a1bba70861 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:10.694283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45566764-9230-4c02-8daa-11a1bba70861 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:10.694540 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45566764-9230-4c02-8daa-11a1bba70861 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:10.696149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-45566764-9230-4c02-8daa-11a1bba70861 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:10.811448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45566764-9230-4c02-8daa-11a1bba70861 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:10.813032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45566764-9230-4c02-8daa-11a1bba70861 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:10.834859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 662/1331] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:10 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:13.379418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604aabc4-7b34-4ef5-973a-ee2493e247a1 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:13.380339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604aabc4-7b34-4ef5-973a-ee2493e247a1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:13.380339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604aabc4-7b34-4ef5-973a-ee2493e247a1 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:13.382553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 670/1332] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:13.391369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65137c4d-179f-486e-b4b4-77c2b85e2cc7 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:13.391808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65137c4d-179f-486e-b4b4-77c2b85e2cc7 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:13.391919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65137c4d-179f-486e-b4b4-77c2b85e2cc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:13.459152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:13.462374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-65137c4d-179f-486e-b4b4-77c2b85e2cc7 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:13.464258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-65137c4d-179f-486e-b4b4-77c2b85e2cc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:13.567822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 663/1333] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:13.863869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604aabc4-7b34-4ef5-973a-ee2493e247a1 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:13.866487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-604aabc4-7b34-4ef5-973a-ee2493e247a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:13.980623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604aabc4-7b34-4ef5-973a-ee2493e247a1 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:14.061127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f98647f-5ce1-4b1d-9aa0-bc7e0c4112a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:14.061736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f98647f-5ce1-4b1d-9aa0-bc7e0c4112a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:14.062004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f98647f-5ce1-4b1d-9aa0-bc7e0c4112a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:14.062331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f98647f-5ce1-4b1d-9aa0-bc7e0c4112a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:14.063913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6f98647f-5ce1-4b1d-9aa0-bc7e0c4112a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:14.094723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:14.184711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f98647f-5ce1-4b1d-9aa0-bc7e0c4112a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:14.186261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f98647f-5ce1-4b1d-9aa0-bc7e0c4112a8 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:14.205728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 671/1334] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:16.389482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816a8f16-b023-4fda-bc60-e08fb48e29d4 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:16.390489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816a8f16-b023-4fda-bc60-e08fb48e29d4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:16.390489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816a8f16-b023-4fda-bc60-e08fb48e29d4 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:16.392523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 664/1335] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:16.400933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6733c6-4242-4a91-98f3-819dbd2785f7 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:16.401391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6733c6-4242-4a91-98f3-819dbd2785f7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:16.401494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6733c6-4242-4a91-98f3-819dbd2785f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:16.468327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-de6733c6-4242-4a91-98f3-819dbd2785f7 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:16.469455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-de6733c6-4242-4a91-98f3-819dbd2785f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:16.546665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:16.569550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 672/1336] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:22:16.850733 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816a8f16-b023-4fda-bc60-e08fb48e29d4 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:16.852298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-816a8f16-b023-4fda-bc60-e08fb48e29d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:16.971581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816a8f16-b023-4fda-bc60-e08fb48e29d4 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:17.003467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:17.052292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d89cad0-d2fe-40f9-b4f9-8ae5e07fb353 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:17.052898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d89cad0-d2fe-40f9-b4f9-8ae5e07fb353 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:17.053171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d89cad0-d2fe-40f9-b4f9-8ae5e07fb353 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:17.053434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d89cad0-d2fe-40f9-b4f9-8ae5e07fb353 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:17.055040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6d89cad0-d2fe-40f9-b4f9-8ae5e07fb353 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:17.181569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d89cad0-d2fe-40f9-b4f9-8ae5e07fb353 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:17.183074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d89cad0-d2fe-40f9-b4f9-8ae5e07fb353 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:17.203288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 665/1337] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:19.875986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ebd2a0-2787-46f6-8128-89d894dd1dc1 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:19.876551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ebd2a0-2787-46f6-8128-89d894dd1dc1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:19.876551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ebd2a0-2787-46f6-8128-89d894dd1dc1 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:19.877761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 673/1338] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:19.884525 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7230c127-652f-499f-8726-f0904d6feb07 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:19.885296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7230c127-652f-499f-8726-f0904d6feb07 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:19.885445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7230c127-652f-499f-8726-f0904d6feb07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:19.959249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7230c127-652f-499f-8726-f0904d6feb07 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:19.960469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7230c127-652f-499f-8726-f0904d6feb07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:19.969077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:20.060058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 666/1339] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:19 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:20.342097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ebd2a0-2787-46f6-8128-89d894dd1dc1 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:20.344244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f0ebd2a0-2787-46f6-8128-89d894dd1dc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:20.465736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ebd2a0-2787-46f6-8128-89d894dd1dc1 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:20.542293 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ed2b6b-5d48-45aa-9ffe-15e092ba01bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:20.542977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ed2b6b-5d48-45aa-9ffe-15e092ba01bf service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:20.543262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ed2b6b-5d48-45aa-9ffe-15e092ba01bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:20.543523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ed2b6b-5d48-45aa-9ffe-15e092ba01bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:20.545115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-94ed2b6b-5d48-45aa-9ffe-15e092ba01bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:20.581631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:20.659551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ed2b6b-5d48-45aa-9ffe-15e092ba01bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:20.660996 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ed2b6b-5d48-45aa-9ffe-15e092ba01bf service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:20.679711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 674/1340] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:20 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:24.914999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8257d-9530-4627-848b-77c15a2650a1 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:24.915543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8257d-9530-4627-848b-77c15a2650a1 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:24.915543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f8257d-9530-4627-848b-77c15a2650a1 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:24.916829 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 667/1341] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:24.922572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2271e05e-0b6c-42d9-91c2-20bd0dafd049 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:24.923341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2271e05e-0b6c-42d9-91c2-20bd0dafd049 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:24.923523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2271e05e-0b6c-42d9-91c2-20bd0dafd049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:25.000166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-2271e05e-0b6c-42d9-91c2-20bd0dafd049 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:25.001438 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2271e05e-0b6c-42d9-91c2-20bd0dafd049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:25.093206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:25.105993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 675/1342] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:25.386237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f8257d-9530-4627-848b-77c15a2650a1 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:25.387814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b9f8257d-9530-4627-848b-77c15a2650a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:25.494896 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f8257d-9530-4627-848b-77c15a2650a1 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:25.534895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:25.573371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacf5d0b-712b-4b17-b307-651f2aa45dd7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:25.574194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf5d0b-712b-4b17-b307-651f2aa45dd7 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:25.574476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf5d0b-712b-4b17-b307-651f2aa45dd7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:25.574753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf5d0b-712b-4b17-b307-651f2aa45dd7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:25.576373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-eacf5d0b-712b-4b17-b307-651f2aa45dd7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:25.691722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacf5d0b-712b-4b17-b307-651f2aa45dd7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:25.693187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacf5d0b-712b-4b17-b307-651f2aa45dd7 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:25.711203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 668/1343] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:28.461022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c705c11-67b4-46b0-ad4e-20798338143a None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:28.461597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c705c11-67b4-46b0-ad4e-20798338143a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:28.461597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c705c11-67b4-46b0-ad4e-20798338143a None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:28.462917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 676/1344] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:28.468520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef7e856-d4e5-4695-a071-af82da1aadcd None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:28.468936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef7e856-d4e5-4695-a071-af82da1aadcd None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:28.469199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef7e856-d4e5-4695-a071-af82da1aadcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:28.538027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7ef7e856-d4e5-4695-a071-af82da1aadcd None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:28.539284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7ef7e856-d4e5-4695-a071-af82da1aadcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:28.557406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:28.645225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 669/1345] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:28 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:28.921107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c705c11-67b4-46b0-ad4e-20798338143a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:28.922705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6c705c11-67b4-46b0-ad4e-20798338143a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:29.038201 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c705c11-67b4-46b0-ad4e-20798338143a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:29.125411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0baa5cc-5fdb-48cf-809a-c390cf0091e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:29.125947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0baa5cc-5fdb-48cf-809a-c390cf0091e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:29.126296 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0baa5cc-5fdb-48cf-809a-c390cf0091e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:29.126703 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0baa5cc-5fdb-48cf-809a-c390cf0091e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:29.129482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c0baa5cc-5fdb-48cf-809a-c390cf0091e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:29.191110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:29.263349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0baa5cc-5fdb-48cf-809a-c390cf0091e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:29.264817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0baa5cc-5fdb-48cf-809a-c390cf0091e3 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:29.284580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 677/1346] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:28 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:33.857519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955637e5-a702-4da2-ab07-4620f75a9b28 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:33.858127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955637e5-a702-4da2-ab07-4620f75a9b28 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:33.858127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955637e5-a702-4da2-ab07-4620f75a9b28 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:33.859460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 670/1347] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:33.865071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae3b4f0-2f77-45eb-bbde-5be7eddae0e6 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:33.865558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae3b4f0-2f77-45eb-bbde-5be7eddae0e6 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:33.865661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae3b4f0-2f77-45eb-bbde-5be7eddae0e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:33.939706 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7ae3b4f0-2f77-45eb-bbde-5be7eddae0e6 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:33.941113 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7ae3b4f0-2f77-45eb-bbde-5be7eddae0e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:34.040310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:34.042660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 678/1348] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:34.324841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955637e5-a702-4da2-ab07-4620f75a9b28 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:34.326446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-955637e5-a702-4da2-ab07-4620f75a9b28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:34.440520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955637e5-a702-4da2-ab07-4620f75a9b28 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:34.489735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:34.523678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76c4c0b-16d6-4870-a2f1-c60d5a271037 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:34.524293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76c4c0b-16d6-4870-a2f1-c60d5a271037 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:34.524568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76c4c0b-16d6-4870-a2f1-c60d5a271037 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:34.524843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76c4c0b-16d6-4870-a2f1-c60d5a271037 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:34.526497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a76c4c0b-16d6-4870-a2f1-c60d5a271037 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:34.642727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76c4c0b-16d6-4870-a2f1-c60d5a271037 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:34.644109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76c4c0b-16d6-4870-a2f1-c60d5a271037 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:34.662933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 671/1349] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:39.632215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a39240-3a3a-48fe-b8a2-e7b301afc7fd None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:39.632771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a39240-3a3a-48fe-b8a2-e7b301afc7fd None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:39.633093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a39240-3a3a-48fe-b8a2-e7b301afc7fd None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:39.635559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 679/1350] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:22:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:39.641718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0ae972-f64c-431a-920b-a1d7657e500c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:39.642240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0ae972-f64c-431a-920b-a1d7657e500c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:39.642434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0ae972-f64c-431a-920b-a1d7657e500c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:39.751653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-4a0ae972-f64c-431a-920b-a1d7657e500c None None] MFA Rules not processed for user `4bc6be7644b24ca9bab23128b41a7148`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:39.753230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4a0ae972-f64c-431a-920b-a1d7657e500c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:39.798083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:39.949804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 672/1351] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:22:39 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:44.210522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc38119-7990-4a97-976e-4d324b40c5cc None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:44.211144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc38119-7990-4a97-976e-4d324b40c5cc None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:44.211144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc38119-7990-4a97-976e-4d324b40c5cc None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:44.212760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 680/1352] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:22:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:44.218437 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a7acf5-d113-46d1-84aa-d1cfe41caaf8 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:44.219000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a7acf5-d113-46d1-84aa-d1cfe41caaf8 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:44.219000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a7acf5-d113-46d1-84aa-d1cfe41caaf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:44.300745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-40a7acf5-d113-46d1-84aa-d1cfe41caaf8 None None] MFA Rules not processed for user `4bc6be7644b24ca9bab23128b41a7148`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:44.301841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-40a7acf5-d113-46d1-84aa-d1cfe41caaf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:44.384970 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:44.423494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 673/1353] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:22:44 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:22:45.543374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8250a32c-53a4-49e6-9500-1adac574556d None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:45.543994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8250a32c-53a4-49e6-9500-1adac574556d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:45.543994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8250a32c-53a4-49e6-9500-1adac574556d None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:45.545509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 681/1354] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:22:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:45.551092 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697a716b-5cfa-4b5e-9c45-7841a72072a5 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:45.551532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697a716b-5cfa-4b5e-9c45-7841a72072a5 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:45.551762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697a716b-5cfa-4b5e-9c45-7841a72072a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:45.632493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-697a716b-5cfa-4b5e-9c45-7841a72072a5 None None] MFA Rules not processed for user `4bc6be7644b24ca9bab23128b41a7148`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:45.633699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-697a716b-5cfa-4b5e-9c45-7841a72072a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:45.743628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 674/1355] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:22:45 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:49.250378 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90cb21d-f4cf-4448-a352-f4275cd8bc22 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:49.250965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90cb21d-f4cf-4448-a352-f4275cd8bc22 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:49.250965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90cb21d-f4cf-4448-a352-f4275cd8bc22 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:49.252171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 682/1356] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:49.257556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d09c518-4599-4256-aae3-ee73ae0f9fe9 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:49.258370 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d09c518-4599-4256-aae3-ee73ae0f9fe9 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:49.258651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d09c518-4599-4256-aae3-ee73ae0f9fe9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:49.300024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:49.334730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2d09c518-4599-4256-aae3-ee73ae0f9fe9 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:49.335935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2d09c518-4599-4256-aae3-ee73ae0f9fe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:49.444172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 675/1357] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:49.751258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79298e7c-7183-4801-8b82-061b94460255 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:49.751539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79298e7c-7183-4801-8b82-061b94460255 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:49.751818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79298e7c-7183-4801-8b82-061b94460255 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:49.753391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 683/1358] 10.222.0.16 () {54 vars in 962 bytes} [Tue Sep 13 15:22:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:49.763410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0d24f9-e793-4003-aaa8-69a96a2f238d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:49.763969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0d24f9-e793-4003-aaa8-69a96a2f238d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:49.764462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0d24f9-e793-4003-aaa8-69a96a2f238d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:49.853870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-ce0d24f9-e793-4003-aaa8-69a96a2f238d None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:49.855456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ce0d24f9-e793-4003-aaa8-69a96a2f238d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:49.911494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:50.002848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 676/1359] 10.222.0.16 () {58 vars in 1059 bytes} [Tue Sep 13 15:22:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:50.011186 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79298e7c-7183-4801-8b82-061b94460255 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:50.012715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-79298e7c-7183-4801-8b82-061b94460255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:50.163392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79298e7c-7183-4801-8b82-061b94460255 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:50.246490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:50.289425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ede4b21-8ffd-47aa-9397-8a4e54b8c283 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:50.290267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ede4b21-8ffd-47aa-9397-8a4e54b8c283 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:50.290513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ede4b21-8ffd-47aa-9397-8a4e54b8c283 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:50.290877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ede4b21-8ffd-47aa-9397-8a4e54b8c283 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:50.292475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8ede4b21-8ffd-47aa-9397-8a4e54b8c283 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:50.399240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ede4b21-8ffd-47aa-9397-8a4e54b8c283 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:50.401447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ede4b21-8ffd-47aa-9397-8a4e54b8c283 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:50.420621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 684/1360] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:52.789868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b70123-c319-4bcd-9132-362acb869cbd None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:52.790521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b70123-c319-4bcd-9132-362acb869cbd None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:52.790521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b70123-c319-4bcd-9132-362acb869cbd None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:52.792331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 677/1361] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:52.797370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61261063-2702-41e4-83ec-a4db20eaa438 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:52.798033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61261063-2702-41e4-83ec-a4db20eaa438 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:52.798033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61261063-2702-41e4-83ec-a4db20eaa438 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:52.868427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-61261063-2702-41e4-83ec-a4db20eaa438 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:52.869659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-61261063-2702-41e4-83ec-a4db20eaa438 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:52.909542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:52.975929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 685/1362] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:53.268449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b70123-c319-4bcd-9132-362acb869cbd None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:53.273243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f7b70123-c319-4bcd-9132-362acb869cbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:53.470955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b70123-c319-4bcd-9132-362acb869cbd None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:53.481574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:53.590065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea4f880-f9f7-4a10-a248-cd9364703e2e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:53.590706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea4f880-f9f7-4a10-a248-cd9364703e2e service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:53.590986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea4f880-f9f7-4a10-a248-cd9364703e2e service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:53.591256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea4f880-f9f7-4a10-a248-cd9364703e2e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:53.592880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8ea4f880-f9f7-4a10-a248-cd9364703e2e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:53.692477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea4f880-f9f7-4a10-a248-cd9364703e2e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:53.694041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea4f880-f9f7-4a10-a248-cd9364703e2e service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:53.712518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 678/1363] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:56.247756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afff6c6-d87a-45d6-baf8-e5e5e6c0914d None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:56.248302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afff6c6-d87a-45d6-baf8-e5e5e6c0914d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:56.248302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afff6c6-d87a-45d6-baf8-e5e5e6c0914d None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:56.249750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 686/1364] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:56.255443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dccb9a8-bc6f-451b-ad00-bf3543748cb5 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:56.256299 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dccb9a8-bc6f-451b-ad00-bf3543748cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:56.256791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dccb9a8-bc6f-451b-ad00-bf3543748cb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:56.273565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:56.333770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8dccb9a8-bc6f-451b-ad00-bf3543748cb5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:56.335561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8dccb9a8-bc6f-451b-ad00-bf3543748cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:56.446245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 679/1365] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:56.734497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addcdf60-03ff-4006-8614-3e2521320b6b None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:56.734748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addcdf60-03ff-4006-8614-3e2521320b6b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:56.735202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addcdf60-03ff-4006-8614-3e2521320b6b None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:56.737187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 687/1366] 10.222.0.16 () {54 vars in 962 bytes} [Tue Sep 13 15:22:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:56.742506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3434bbfb-1427-4999-b927-0fe4a0eed988 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:56.742902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3434bbfb-1427-4999-b927-0fe4a0eed988 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:56.743237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3434bbfb-1427-4999-b927-0fe4a0eed988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:56.825159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3434bbfb-1427-4999-b927-0fe4a0eed988 None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:56.826430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3434bbfb-1427-4999-b927-0fe4a0eed988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:56.866678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:56.957822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 680/1367] 10.222.0.16 () {58 vars in 1059 bytes} [Tue Sep 13 15:22:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:22:56.966920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addcdf60-03ff-4006-8614-3e2521320b6b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:56.968684 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-addcdf60-03ff-4006-8614-3e2521320b6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:57.126903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addcdf60-03ff-4006-8614-3e2521320b6b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:57.147964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:57.250816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59db8f8-d7a3-42be-9949-13254290b932 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:57.251477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59db8f8-d7a3-42be-9949-13254290b932 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:57.251583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59db8f8-d7a3-42be-9949-13254290b932 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:57.251859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59db8f8-d7a3-42be-9949-13254290b932 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:57.253606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f59db8f8-d7a3-42be-9949-13254290b932 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:57.357532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59db8f8-d7a3-42be-9949-13254290b932 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:57.359534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59db8f8-d7a3-42be-9949-13254290b932 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:57.367740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb1c811-9416-4bc0-ac77-97b510c0de2d None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:57.368222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb1c811-9416-4bc0-ac77-97b510c0de2d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:57.368437 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb1c811-9416-4bc0-ac77-97b510c0de2d None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:57.370050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 681/1368] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:57.374908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54b175-a1c6-4cf5-93c1-c4613037be34 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:57.375248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54b175-a1c6-4cf5-93c1-c4613037be34 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:57.375576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f54b175-a1c6-4cf5-93c1-c4613037be34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:57.384362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 688/1369] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:57.450131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-0f54b175-a1c6-4cf5-93c1-c4613037be34 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:57.451557 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0f54b175-a1c6-4cf5-93c1-c4613037be34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:57.582640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:57.593018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 682/1370] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:22:57.947837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85430b9f-9e61-45c8-95b7-0d10dbe613c8 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:57.947837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85430b9f-9e61-45c8-95b7-0d10dbe613c8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:57.947837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85430b9f-9e61-45c8-95b7-0d10dbe613c8 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:57.948934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 689/1371] 10.222.0.16 () {54 vars in 964 bytes} [Tue Sep 13 15:22:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:57.953231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bea6c4-0803-42e1-aa3e-252ae02ce9df None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:57.953615 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bea6c4-0803-42e1-aa3e-252ae02ce9df None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:57.953987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bea6c4-0803-42e1-aa3e-252ae02ce9df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:58.045146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-f6bea6c4-0803-42e1-aa3e-252ae02ce9df None None] MFA Rules not processed for user `39c54901c5b5400f96e084fd4adf54e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:58.046636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f6bea6c4-0803-42e1-aa3e-252ae02ce9df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:58.132258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 683/1372] 10.222.0.16 () {58 vars in 1061 bytes} [Tue Sep 13 15:22:57 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:22:58.140957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85430b9f-9e61-45c8-95b7-0d10dbe613c8 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:22:58.142571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-85430b9f-9e61-45c8-95b7-0d10dbe613c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:58.175919 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:58.269833 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85430b9f-9e61-45c8-95b7-0d10dbe613c8 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:22:58.360688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcf700c-adfe-4329-800a-cec8d8171b77 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:22:58.361474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcf700c-adfe-4329-800a-cec8d8171b77 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:58.361769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcf700c-adfe-4329-800a-cec8d8171b77 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:58.362196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcf700c-adfe-4329-800a-cec8d8171b77 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:58.363946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0fcf700c-adfe-4329-800a-cec8d8171b77 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:58.451303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:22:58.487750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcf700c-adfe-4329-800a-cec8d8171b77 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:22:58.489230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcf700c-adfe-4329-800a-cec8d8171b77 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:22:58.509676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 690/1373] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:22:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:22:59.609351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf231a44-233f-4adf-b0a5-5970e8f58613 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:59.609895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf231a44-233f-4adf-b0a5-5970e8f58613 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:59.609895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf231a44-233f-4adf-b0a5-5970e8f58613 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:59.611388 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 684/1374] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:22:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:22:59.616697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da28329c-52ae-4e48-a033-41e15f9e8046 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:22:59.617141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da28329c-52ae-4e48-a033-41e15f9e8046 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:22:59.617251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da28329c-52ae-4e48-a033-41e15f9e8046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:22:59.691210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-da28329c-52ae-4e48-a033-41e15f9e8046 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:22:59.692427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-da28329c-52ae-4e48-a033-41e15f9e8046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:22:59.798432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 691/1375] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:22:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:00.089420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf231a44-233f-4adf-b0a5-5970e8f58613 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:00.091018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-bf231a44-233f-4adf-b0a5-5970e8f58613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:00.184953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:00.247772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf231a44-233f-4adf-b0a5-5970e8f58613 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:00.366295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acd0911-0cb1-4b79-92dd-37b8b1ae40d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:00.366922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd0911-0cb1-4b79-92dd-37b8b1ae40d9 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:00.367225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd0911-0cb1-4b79-92dd-37b8b1ae40d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:00.367510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd0911-0cb1-4b79-92dd-37b8b1ae40d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:00.369097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3acd0911-0cb1-4b79-92dd-37b8b1ae40d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:00.423889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:00.466763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd0911-0cb1-4b79-92dd-37b8b1ae40d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:00.468385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd0911-0cb1-4b79-92dd-37b8b1ae40d9 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:00.488267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 685/1376] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:02.728341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fdad88-5c24-4679-9f7d-f04d197254b9 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:02.729248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fdad88-5c24-4679-9f7d-f04d197254b9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:02.729497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fdad88-5c24-4679-9f7d-f04d197254b9 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:02.731087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 692/1377] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:23:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:02.736457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43973c1-58f3-4323-a5ee-eac164768684 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:02.736912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43973c1-58f3-4323-a5ee-eac164768684 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:02.737015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43973c1-58f3-4323-a5ee-eac164768684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:02.814638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-b43973c1-58f3-4323-a5ee-eac164768684 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:02.815851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b43973c1-58f3-4323-a5ee-eac164768684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:02.922156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 686/1378] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:23:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:03.210285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fdad88-5c24-4679-9f7d-f04d197254b9 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:03.212411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f4fdad88-5c24-4679-9f7d-f04d197254b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:03.258384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:03.383015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fdad88-5c24-4679-9f7d-f04d197254b9 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:03.495160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:03.500499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5debd4-3bc6-48c1-8984-0075ca5a3997 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:03.501133 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5debd4-3bc6-48c1-8984-0075ca5a3997 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:03.501393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5debd4-3bc6-48c1-8984-0075ca5a3997 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:03.502080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5debd4-3bc6-48c1-8984-0075ca5a3997 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:03.504498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2c5debd4-3bc6-48c1-8984-0075ca5a3997 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:03.602592 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5debd4-3bc6-48c1-8984-0075ca5a3997 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:03.604204 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5debd4-3bc6-48c1-8984-0075ca5a3997 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:03.623527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 693/1379] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:05.861697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b632e925-c327-466f-a9f4-b67337be6766 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:05.862296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b632e925-c327-466f-a9f4-b67337be6766 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:05.862296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b632e925-c327-466f-a9f4-b67337be6766 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:05.864994 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 687/1380] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:23:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:05.870144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d619feb3-d3b9-49f5-8a09-1c4a029d583b None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:05.870708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d619feb3-d3b9-49f5-8a09-1c4a029d583b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:05.870708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d619feb3-d3b9-49f5-8a09-1c4a029d583b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:05.965136 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d619feb3-d3b9-49f5-8a09-1c4a029d583b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:05.966507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d619feb3-d3b9-49f5-8a09-1c4a029d583b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:06.037583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:06.075412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 694/1381] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:23:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:06.363140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b632e925-c327-466f-a9f4-b67337be6766 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:06.364811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b632e925-c327-466f-a9f4-b67337be6766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:06.394517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:06.518987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b632e925-c327-466f-a9f4-b67337be6766 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:06.633051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:06.653900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac09983-d469-4c1b-938a-6d29bf341614 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:06.654531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac09983-d469-4c1b-938a-6d29bf341614 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:06.654776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac09983-d469-4c1b-938a-6d29bf341614 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:06.655042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac09983-d469-4c1b-938a-6d29bf341614 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:06.656713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4ac09983-d469-4c1b-938a-6d29bf341614 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:06.874626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac09983-d469-4c1b-938a-6d29bf341614 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:06.876745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac09983-d469-4c1b-938a-6d29bf341614 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:06.901051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 688/1382] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 538 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:09.151930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b5a2d-911d-4cc2-af1b-e8cb69cf9ee6 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:09.152485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b5a2d-911d-4cc2-af1b-e8cb69cf9ee6 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:09.152485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4b5a2d-911d-4cc2-af1b-e8cb69cf9ee6 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:09.154132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 695/1383] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:23:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:23:09.160024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96112f2d-5db5-4885-bc8f-07d486faa7a8 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:09.160486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96112f2d-5db5-4885-bc8f-07d486faa7a8 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:09.160576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96112f2d-5db5-4885-bc8f-07d486faa7a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:09.242667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-96112f2d-5db5-4885-bc8f-07d486faa7a8 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:09.243964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-96112f2d-5db5-4885-bc8f-07d486faa7a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:09.311980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:09.361590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 689/1384] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:23:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:09.649148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4b5a2d-911d-4cc2-af1b-e8cb69cf9ee6 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:09.651206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2f4b5a2d-911d-4cc2-af1b-e8cb69cf9ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:09.827172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4b5a2d-911d-4cc2-af1b-e8cb69cf9ee6 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:09.888387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:09.960475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b51017-aff0-485d-beac-9d68b6fee77b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:09.961179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b51017-aff0-485d-beac-9d68b6fee77b service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:09.961465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b51017-aff0-485d-beac-9d68b6fee77b service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:09.961727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b51017-aff0-485d-beac-9d68b6fee77b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:09.963496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-80b51017-aff0-485d-beac-9d68b6fee77b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:10.059512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b51017-aff0-485d-beac-9d68b6fee77b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:10.061138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b51017-aff0-485d-beac-9d68b6fee77b service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:10.082133 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 696/1385] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:12.319192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083e2fda-ce09-4378-b5ad-a8e986e49e85 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:12.319850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083e2fda-ce09-4378-b5ad-a8e986e49e85 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:12.319850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083e2fda-ce09-4378-b5ad-a8e986e49e85 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:12.321234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 690/1386] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:23:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:12.326831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad563a-5ea0-4d74-8a69-eb5157880d1e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:12.327284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad563a-5ea0-4d74-8a69-eb5157880d1e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:12.327398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad563a-5ea0-4d74-8a69-eb5157880d1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:12.395921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d4ad563a-5ea0-4d74-8a69-eb5157880d1e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:12.397140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d4ad563a-5ea0-4d74-8a69-eb5157880d1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:12.406668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:12.624701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 697/1387] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:23:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:12.911673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083e2fda-ce09-4378-b5ad-a8e986e49e85 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:12.913209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-083e2fda-ce09-4378-b5ad-a8e986e49e85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:13.065488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083e2fda-ce09-4378-b5ad-a8e986e49e85 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:13.117425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:13.186584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76afe9a5-2bf6-4bab-99a8-7c11fbf00d54 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:13.187102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76afe9a5-2bf6-4bab-99a8-7c11fbf00d54 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:13.187374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76afe9a5-2bf6-4bab-99a8-7c11fbf00d54 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:13.187638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76afe9a5-2bf6-4bab-99a8-7c11fbf00d54 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:13.189215 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-76afe9a5-2bf6-4bab-99a8-7c11fbf00d54 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:13.280632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76afe9a5-2bf6-4bab-99a8-7c11fbf00d54 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:13.282260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76afe9a5-2bf6-4bab-99a8-7c11fbf00d54 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:13.300692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 691/1388] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:13.778820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac4c9e4-e9ce-4d8a-8fb1-7fe0a2ca1100 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:13.779394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac4c9e4-e9ce-4d8a-8fb1-7fe0a2ca1100 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:13.779394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac4c9e4-e9ce-4d8a-8fb1-7fe0a2ca1100 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:13.780744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 698/1389] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:23:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:13.786166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9354acd-d15d-44fb-a783-2090b1539224 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:13.786588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9354acd-d15d-44fb-a783-2090b1539224 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:13.786766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9354acd-d15d-44fb-a783-2090b1539224 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:13.854942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:13.858928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d9354acd-d15d-44fb-a783-2090b1539224 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:13.860865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d9354acd-d15d-44fb-a783-2090b1539224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:13.961514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 692/1390] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:23:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:14.259902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edc1d9a-7a5b-4551-ba5b-989a74183099 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:14.260159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edc1d9a-7a5b-4551-ba5b-989a74183099 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:14.260425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edc1d9a-7a5b-4551-ba5b-989a74183099 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:14.261789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 699/1391] 10.222.0.16 () {54 vars in 966 bytes} [Tue Sep 13 15:23:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:14.268497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcb851f-f2e4-46bb-aa7c-33b33ae91c5a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:14.268763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcb851f-f2e4-46bb-aa7c-33b33ae91c5a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:14.269072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcb851f-f2e4-46bb-aa7c-33b33ae91c5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:14.354717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-0fcb851f-f2e4-46bb-aa7c-33b33ae91c5a None None] MFA Rules not processed for user `2ce646dcb8124685ac7a1c1eca664d04`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:14.355958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0fcb851f-f2e4-46bb-aa7c-33b33ae91c5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:14.428968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:14.435679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 693/1392] 10.222.0.16 () {58 vars in 1063 bytes} [Tue Sep 13 15:23:14 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:14.445571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edc1d9a-7a5b-4551-ba5b-989a74183099 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:14.447135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2edc1d9a-7a5b-4551-ba5b-989a74183099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:14.560330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edc1d9a-7a5b-4551-ba5b-989a74183099 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:14.599614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:14.637040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4ac945-6267-4d8b-9ddf-9cfe9b42a9d8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce646dcb8124685ac7a1c1eca664d04', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:14.637661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4ac945-6267-4d8b-9ddf-9cfe9b42a9d8 service glance] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:14.637933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4ac945-6267-4d8b-9ddf-9cfe9b42a9d8 service glance] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:14.638238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4ac945-6267-4d8b-9ddf-9cfe9b42a9d8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:14.639852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0d4ac945-6267-4d8b-9ddf-9cfe9b42a9d8 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:14.753597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4ac945-6267-4d8b-9ddf-9cfe9b42a9d8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:14.755348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4ac945-6267-4d8b-9ddf-9cfe9b42a9d8 service glance] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:14.773542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 700/1393] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:15.528742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ae123-415a-4b67-b688-e1a9acd2e763 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:15.529195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ae123-415a-4b67-b688-e1a9acd2e763 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:15.529195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ae123-415a-4b67-b688-e1a9acd2e763 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:15.530542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 694/1394] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:23:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:23:15.535846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273fdd39-fcd7-4821-bca3-4c474e6ae018 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:15.536262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273fdd39-fcd7-4821-bca3-4c474e6ae018 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:15.536428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273fdd39-fcd7-4821-bca3-4c474e6ae018 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:15.539707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0a9df9-7b08-4781-a1f9-3a709d2f0162 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:15.539986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0a9df9-7b08-4781-a1f9-3a709d2f0162 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:15.540254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0a9df9-7b08-4781-a1f9-3a709d2f0162 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:15.608774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-273fdd39-fcd7-4821-bca3-4c474e6ae018 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:15.609991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-ae0a9df9-7b08-4781-a1f9-3a709d2f0162 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:15.610180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-273fdd39-fcd7-4821-bca3-4c474e6ae018 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:15.611242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ae0a9df9-7b08-4781-a1f9-3a709d2f0162 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:15.628937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:15.704149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 695/1395] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:23:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:15.714866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 701/1396] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:23:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:15.749510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0865e691-e82d-44a5-9354-048740c1c682 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:15.749796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0865e691-e82d-44a5-9354-048740c1c682 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:15.750071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0865e691-e82d-44a5-9354-048740c1c682 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:15.751479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 696/1397] 10.222.0.16 () {54 vars in 943 bytes} [Tue Sep 13 15:23:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:15.757880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817b814-4b35-439e-beb7-ee26fbc925f0 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:15.761936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817b814-4b35-439e-beb7-ee26fbc925f0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:15.762262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817b814-4b35-439e-beb7-ee26fbc925f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:15.846567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b817b814-4b35-439e-beb7-ee26fbc925f0 None None] MFA Rules not processed for user `35ece665f492449282ab03ad91f0b254`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:15.847823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b817b814-4b35-439e-beb7-ee26fbc925f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:15.930410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:15.956957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 702/1398] 10.222.0.16 () {58 vars in 1040 bytes} [Tue Sep 13 15:23:15 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:15.966528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0865e691-e82d-44a5-9354-048740c1c682 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:15.968117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0865e691-e82d-44a5-9354-048740c1c682 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:16.015419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817b814-4b35-439e-beb7-ee26fbc925f0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:16.017005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b817b814-4b35-439e-beb7-ee26fbc925f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:16.056161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:16.110342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0865e691-e82d-44a5-9354-048740c1c682 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:16.121837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817b814-4b35-439e-beb7-ee26fbc925f0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:16.220902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4561c16-3b54-47a9-a158-23a995d91507 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:16.221866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4561c16-3b54-47a9-a158-23a995d91507 service swift] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:16.222526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4561c16-3b54-47a9-a158-23a995d91507 service swift] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:16.222797 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4561c16-3b54-47a9-a158-23a995d91507 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:16.224449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d4561c16-3b54-47a9-a158-23a995d91507 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:16.229638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:16.240844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b2aed-1aab-4028-b9b2-c6e53cce2d30 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:16.241457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b2aed-1aab-4028-b9b2-c6e53cce2d30 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:16.241760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b2aed-1aab-4028-b9b2-c6e53cce2d30 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:16.242023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b2aed-1aab-4028-b9b2-c6e53cce2d30 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:16.243816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ef9b2aed-1aab-4028-b9b2-c6e53cce2d30 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:16.294093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:16.312051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4561c16-3b54-47a9-a158-23a995d91507 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:16.313649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4561c16-3b54-47a9-a158-23a995d91507 service swift] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:16.321768 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 697/1399] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:23:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:23:16.337959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9b2aed-1aab-4028-b9b2-c6e53cce2d30 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:16.339573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9b2aed-1aab-4028-b9b2-c6e53cce2d30 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:16.362041 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 703/1400] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:18.575406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9818504-2d25-4c2b-8c50-306469c7db65 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:18.575984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9818504-2d25-4c2b-8c50-306469c7db65 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:18.575984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9818504-2d25-4c2b-8c50-306469c7db65 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:18.578392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 698/1401] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:23:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:18.582828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec18849-7a60-4933-a402-c117b9f8fd94 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:18.583482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec18849-7a60-4933-a402-c117b9f8fd94 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:18.583482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec18849-7a60-4933-a402-c117b9f8fd94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:18.658273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-9ec18849-7a60-4933-a402-c117b9f8fd94 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:18.659601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9ec18849-7a60-4933-a402-c117b9f8fd94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:18.732217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:18.773521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 704/1402] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:23:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:19.060592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9818504-2d25-4c2b-8c50-306469c7db65 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:19.062196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e9818504-2d25-4c2b-8c50-306469c7db65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:19.225624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9818504-2d25-4c2b-8c50-306469c7db65 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:19.288892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:19.357684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7093698-bea1-467e-b18e-dae395a56f41 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:19.358338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7093698-bea1-467e-b18e-dae395a56f41 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:19.358613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7093698-bea1-467e-b18e-dae395a56f41 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:19.358870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7093698-bea1-467e-b18e-dae395a56f41 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:19.363111 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e7093698-bea1-467e-b18e-dae395a56f41 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:19.466731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7093698-bea1-467e-b18e-dae395a56f41 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:19.468431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7093698-bea1-467e-b18e-dae395a56f41 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:19.489507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 699/1403] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:21.722827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6e8cc-06f4-410a-91c1-bcd9b7284990 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:21.723432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6e8cc-06f4-410a-91c1-bcd9b7284990 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:21.723432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6e8cc-06f4-410a-91c1-bcd9b7284990 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:21.729219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 705/1404] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:23:21 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:21.730982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26f3dc-df9f-4924-8c3f-1b3a5285656b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:21.731436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26f3dc-df9f-4924-8c3f-1b3a5285656b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:21.731545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26f3dc-df9f-4924-8c3f-1b3a5285656b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:21.808214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7a26f3dc-df9f-4924-8c3f-1b3a5285656b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:21.809433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7a26f3dc-df9f-4924-8c3f-1b3a5285656b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:21.832711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:21.918345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 700/1405] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:23:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:22.205549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c6e8cc-06f4-410a-91c1-bcd9b7284990 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:22.207126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-57c6e8cc-06f4-410a-91c1-bcd9b7284990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:22.420009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c6e8cc-06f4-410a-91c1-bcd9b7284990 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:22.500856 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:22.591161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49b9b15-1e6a-484a-a0aa-f92ad27ac0d6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:22.591785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49b9b15-1e6a-484a-a0aa-f92ad27ac0d6 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:22.592066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49b9b15-1e6a-484a-a0aa-f92ad27ac0d6 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:22.592342 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49b9b15-1e6a-484a-a0aa-f92ad27ac0d6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:22.593962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b49b9b15-1e6a-484a-a0aa-f92ad27ac0d6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:22.757143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49b9b15-1e6a-484a-a0aa-f92ad27ac0d6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:22.758893 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49b9b15-1e6a-484a-a0aa-f92ad27ac0d6 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:22.779531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 706/1406] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 574 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:24.974944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bb4bc5-3411-48e0-9959-fd305adea87e None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:24.975459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bb4bc5-3411-48e0-9959-fd305adea87e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:24.975459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bb4bc5-3411-48e0-9959-fd305adea87e None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:24.976994 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 701/1407] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:23:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:24.982126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f61293f-3e44-440e-aa1e-60be21eb597a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:24.982577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f61293f-3e44-440e-aa1e-60be21eb597a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:24.982662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f61293f-3e44-440e-aa1e-60be21eb597a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:25.050586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-6f61293f-3e44-440e-aa1e-60be21eb597a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:25.051791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6f61293f-3e44-440e-aa1e-60be21eb597a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:25.054867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:25.163558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 707/1408] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:23:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:25.447799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bb4bc5-3411-48e0-9959-fd305adea87e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:25.449339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d1bb4bc5-3411-48e0-9959-fd305adea87e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:25.619224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bb4bc5-3411-48e0-9959-fd305adea87e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:25.623066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:25.746749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12716582-2fe8-422d-b2f5-50820ddc0498 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:25.747267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12716582-2fe8-422d-b2f5-50820ddc0498 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:25.747538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12716582-2fe8-422d-b2f5-50820ddc0498 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:25.747801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12716582-2fe8-422d-b2f5-50820ddc0498 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:25.749403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-12716582-2fe8-422d-b2f5-50820ddc0498 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:25.845521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12716582-2fe8-422d-b2f5-50820ddc0498 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:25.847610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12716582-2fe8-422d-b2f5-50820ddc0498 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:25.868577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 702/1409] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:35.161414 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=41 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:35.865774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:38.885663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b26018-4b45-4009-b290-47bc5352d1f9 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:38.886186 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b26018-4b45-4009-b290-47bc5352d1f9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:38.886287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b26018-4b45-4009-b290-47bc5352d1f9 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:38.887616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 708/1410] 10.222.0.16 () {54 vars in 918 bytes} [Tue Sep 13 15:23:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:38.893066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0021ea0-df4b-4416-9ba2-8bdf0356bd66 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:38.893550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0021ea0-df4b-4416-9ba2-8bdf0356bd66 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:38.893657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0021ea0-df4b-4416-9ba2-8bdf0356bd66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:38.971282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-a0021ea0-df4b-4416-9ba2-8bdf0356bd66 None None] MFA Rules not processed for user `4bc6be7644b24ca9bab23128b41a7148`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:38.972844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a0021ea0-df4b-4416-9ba2-8bdf0356bd66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:39.079924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 703/1411] 10.222.0.16 () {58 vars in 1015 bytes} [Tue Sep 13 15:23:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:23:39.095200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4020e25-69f7-46a7-9d60-3376d5d8b474 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:39.095478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4020e25-69f7-46a7-9d60-3376d5d8b474 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:39.095742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4020e25-69f7-46a7-9d60-3376d5d8b474 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:39.097073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 709/1412] 10.222.0.16 () {54 vars in 918 bytes} [Tue Sep 13 15:23:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:39.102426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413d278f-5d91-4d57-a44e-ece7cf55458d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:39.102711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413d278f-5d91-4d57-a44e-ece7cf55458d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:39.102979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413d278f-5d91-4d57-a44e-ece7cf55458d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:39.197587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:39.200725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-413d278f-5d91-4d57-a44e-ece7cf55458d None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:39.202359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-413d278f-5d91-4d57-a44e-ece7cf55458d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:39.324622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 704/1413] 10.222.0.16 () {58 vars in 1015 bytes} [Tue Sep 13 15:23:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:41.910620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79c734f-b8b3-4d3f-8a26-073a39a057d4 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:41.911497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79c734f-b8b3-4d3f-8a26-073a39a057d4 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:41.911497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79c734f-b8b3-4d3f-8a26-073a39a057d4 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:41.913588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 710/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:23:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:41.919387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c908a05d-f635-4935-ab6b-ca9f449af05b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:41.919833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c908a05d-f635-4935-ab6b-ca9f449af05b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:41.919937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c908a05d-f635-4935-ab6b-ca9f449af05b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:41.988312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c908a05d-f635-4935-ab6b-ca9f449af05b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:41.989506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c908a05d-f635-4935-ab6b-ca9f449af05b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:42.056946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:42.100746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 705/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:23:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:42.394163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79c734f-b8b3-4d3f-8a26-073a39a057d4 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:42.396147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a79c734f-b8b3-4d3f-8a26-073a39a057d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:42.562494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79c734f-b8b3-4d3f-8a26-073a39a057d4 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:42.654715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:42.696582 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e4a36b-bd38-4d1d-a3d2-3fe70ddf9041 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:42.697203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e4a36b-bd38-4d1d-a3d2-3fe70ddf9041 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:42.697478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e4a36b-bd38-4d1d-a3d2-3fe70ddf9041 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:42.697747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e4a36b-bd38-4d1d-a3d2-3fe70ddf9041 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:42.699411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-14e4a36b-bd38-4d1d-a3d2-3fe70ddf9041 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:42.805569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e4a36b-bd38-4d1d-a3d2-3fe70ddf9041 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:42.807185 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e4a36b-bd38-4d1d-a3d2-3fe70ddf9041 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:42.825460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 711/1416] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:44.002658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de49b688-2a04-4869-9023-c0f5ae126f3b None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:44.003292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de49b688-2a04-4869-9023-c0f5ae126f3b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:44.003292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de49b688-2a04-4869-9023-c0f5ae126f3b None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:44.004538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 706/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:23:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:44.012089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f57be4-d9cf-489d-bc68-9076084db52c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:44.012786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f57be4-d9cf-489d-bc68-9076084db52c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:44.012950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f57be4-d9cf-489d-bc68-9076084db52c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:44.103335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-75f57be4-d9cf-489d-bc68-9076084db52c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:44.104545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-75f57be4-d9cf-489d-bc68-9076084db52c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:44.132420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:44.207731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 712/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:23:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:44.356995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de49b688-2a04-4869-9023-c0f5ae126f3b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:44.359197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-de49b688-2a04-4869-9023-c0f5ae126f3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:44.521760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de49b688-2a04-4869-9023-c0f5ae126f3b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:44.576554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:44.659969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943248b4-a21a-47ac-921b-65493feb3044 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:44.660583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943248b4-a21a-47ac-921b-65493feb3044 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:44.660855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943248b4-a21a-47ac-921b-65493feb3044 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:44.661120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943248b4-a21a-47ac-921b-65493feb3044 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:44.662752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-943248b4-a21a-47ac-921b-65493feb3044 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:44.759453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943248b4-a21a-47ac-921b-65493feb3044 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:44.761071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943248b4-a21a-47ac-921b-65493feb3044 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:44.786685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 707/1419] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:45.930730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb6956-7da8-4f8a-9cd2-cac10864c4e8 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:45.931337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb6956-7da8-4f8a-9cd2-cac10864c4e8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:45.931337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb6956-7da8-4f8a-9cd2-cac10864c4e8 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:45.932705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 713/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:23:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:45.940577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca5667-4fd8-4777-8d1d-b0cd7e1d02fd None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:45.941048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca5667-4fd8-4777-8d1d-b0cd7e1d02fd None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:45.941154 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca5667-4fd8-4777-8d1d-b0cd7e1d02fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:46.006937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:46.035081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-69ca5667-4fd8-4777-8d1d-b0cd7e1d02fd None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:46.036274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-69ca5667-4fd8-4777-8d1d-b0cd7e1d02fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:46.146315 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 708/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:23:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:46.296770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7eb6956-7da8-4f8a-9cd2-cac10864c4e8 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:46.298341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a7eb6956-7da8-4f8a-9cd2-cac10864c4e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:46.466164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7eb6956-7da8-4f8a-9cd2-cac10864c4e8 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:46.478961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:46.585054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5d268f-4dae-4331-8815-17276f3dcbe8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:46.585672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5d268f-4dae-4331-8815-17276f3dcbe8 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:46.585948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5d268f-4dae-4331-8815-17276f3dcbe8 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:46.586252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5d268f-4dae-4331-8815-17276f3dcbe8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:46.587984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9f5d268f-4dae-4331-8815-17276f3dcbe8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:46.689248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5d268f-4dae-4331-8815-17276f3dcbe8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:46.690900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5d268f-4dae-4331-8815-17276f3dcbe8 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:46.712570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 714/1422] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:47.811279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539dc49b-3274-43cb-97d8-71bee638c481 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:47.811870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539dc49b-3274-43cb-97d8-71bee638c481 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:47.811870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539dc49b-3274-43cb-97d8-71bee638c481 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:47.813140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 709/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:23:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:47.820613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d84aa8-34f0-4c86-bdaa-121353a6b905 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:47.821388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d84aa8-34f0-4c86-bdaa-121353a6b905 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:47.822144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d84aa8-34f0-4c86-bdaa-121353a6b905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:47.842850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:47.914259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-28d84aa8-34f0-4c86-bdaa-121353a6b905 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:47.915646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-28d84aa8-34f0-4c86-bdaa-121353a6b905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:48.024385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 715/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:48.178336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539dc49b-3274-43cb-97d8-71bee638c481 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:48.180672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-539dc49b-3274-43cb-97d8-71bee638c481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:48.302613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:48.351614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539dc49b-3274-43cb-97d8-71bee638c481 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:48.472314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52a0b2d-26a2-4b05-a33b-1ed8072726dd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:48.473158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52a0b2d-26a2-4b05-a33b-1ed8072726dd service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:48.473463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52a0b2d-26a2-4b05-a33b-1ed8072726dd service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:48.473735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52a0b2d-26a2-4b05-a33b-1ed8072726dd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:48.475377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e52a0b2d-26a2-4b05-a33b-1ed8072726dd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:48.562365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:48.577020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52a0b2d-26a2-4b05-a33b-1ed8072726dd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:48.578667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52a0b2d-26a2-4b05-a33b-1ed8072726dd service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:48.596791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 710/1425] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:49.706465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e38320-d42e-46be-9aaf-90b8349fadd4 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:49.707369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e38320-d42e-46be-9aaf-90b8349fadd4 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:49.707369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e38320-d42e-46be-9aaf-90b8349fadd4 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:49.709582 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 716/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:23:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:49.716233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b43b27-cdda-460e-bfbe-8ecaf7bd9b6a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:49.716686 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b43b27-cdda-460e-bfbe-8ecaf7bd9b6a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:49.716803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b43b27-cdda-460e-bfbe-8ecaf7bd9b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:49.796098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-35b43b27-cdda-460e-bfbe-8ecaf7bd9b6a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:49.797226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-35b43b27-cdda-460e-bfbe-8ecaf7bd9b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:49.911765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 711/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:23:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:50.066235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e38320-d42e-46be-9aaf-90b8349fadd4 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:50.067712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-68e38320-d42e-46be-9aaf-90b8349fadd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:50.156875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:50.220314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e38320-d42e-46be-9aaf-90b8349fadd4 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:50.344472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63a2cee-c884-401e-8016-390ea7e04785 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:50.345021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63a2cee-c884-401e-8016-390ea7e04785 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:50.345298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63a2cee-c884-401e-8016-390ea7e04785 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:50.345565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63a2cee-c884-401e-8016-390ea7e04785 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:50.347213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a63a2cee-c884-401e-8016-390ea7e04785 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:50.405561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:50.447016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63a2cee-c884-401e-8016-390ea7e04785 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:50.448658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63a2cee-c884-401e-8016-390ea7e04785 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:50.466529 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 717/1428] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:51.525527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ffdb30-1653-4c11-846e-c9518178c0f2 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:51.526403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ffdb30-1653-4c11-846e-c9518178c0f2 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:51.526403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ffdb30-1653-4c11-846e-c9518178c0f2 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:51.528478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 712/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:23:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:51.538309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4444ea09-d932-4f7d-9b26-c4c411bc87f8 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:51.538747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4444ea09-d932-4f7d-9b26-c4c411bc87f8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:51.538872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4444ea09-d932-4f7d-9b26-c4c411bc87f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:51.629319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4444ea09-d932-4f7d-9b26-c4c411bc87f8 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:51.630596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4444ea09-d932-4f7d-9b26-c4c411bc87f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:51.735251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 718/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:23:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:51.888180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ffdb30-1653-4c11-846e-c9518178c0f2 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:51.891141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b2ffdb30-1653-4c11-846e-c9518178c0f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:51.945326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:52.045079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ffdb30-1653-4c11-846e-c9518178c0f2 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:52.171351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e83fd5-97a4-47cb-8aac-89783cc9cea0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:52.171965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e83fd5-97a4-47cb-8aac-89783cc9cea0 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:52.172242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e83fd5-97a4-47cb-8aac-89783cc9cea0 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:52.172510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e83fd5-97a4-47cb-8aac-89783cc9cea0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:52.174209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-74e83fd5-97a4-47cb-8aac-89783cc9cea0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:52.181309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:52.272633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e83fd5-97a4-47cb-8aac-89783cc9cea0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:52.274640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e83fd5-97a4-47cb-8aac-89783cc9cea0 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:52.294057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 713/1431] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:53.390317 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018f370-c6a2-40a3-98e2-66e09975e152 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:53.391572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018f370-c6a2-40a3-98e2-66e09975e152 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:53.391572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e018f370-c6a2-40a3-98e2-66e09975e152 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:53.393272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 719/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:23:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:53.402244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d141587c-2f89-46a3-bd14-042dc3dafbad None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:53.402767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d141587c-2f89-46a3-bd14-042dc3dafbad None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:53.402767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d141587c-2f89-46a3-bd14-042dc3dafbad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:53.483808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d141587c-2f89-46a3-bd14-042dc3dafbad None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:53.485021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d141587c-2f89-46a3-bd14-042dc3dafbad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:53.556774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:53.588207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 714/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:23:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:53.740816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e018f370-c6a2-40a3-98e2-66e09975e152 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:53.742425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e018f370-c6a2-40a3-98e2-66e09975e152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:53.761597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:53.896639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e018f370-c6a2-40a3-98e2-66e09975e152 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:53.985256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:54.012377 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b78be2-d046-4798-90ef-51e1fc52c589 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:54.013135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b78be2-d046-4798-90ef-51e1fc52c589 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:54.013414 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b78be2-d046-4798-90ef-51e1fc52c589 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:54.013684 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b78be2-d046-4798-90ef-51e1fc52c589 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:54.015323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-22b78be2-d046-4798-90ef-51e1fc52c589 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:54.111217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b78be2-d046-4798-90ef-51e1fc52c589 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:54.112814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b78be2-d046-4798-90ef-51e1fc52c589 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:54.130902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 720/1434] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:55.199502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19da614a-27cf-4cbb-8257-c699bffeb940 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:55.200107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19da614a-27cf-4cbb-8257-c699bffeb940 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:55.200107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19da614a-27cf-4cbb-8257-c699bffeb940 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:55.201394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 715/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:23:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:55.208741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef36bac5-32a7-4207-a0ab-0383e28b69ae None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:55.209487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef36bac5-32a7-4207-a0ab-0383e28b69ae None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:55.209636 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef36bac5-32a7-4207-a0ab-0383e28b69ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:55.290750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-ef36bac5-32a7-4207-a0ab-0383e28b69ae None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:55.291975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ef36bac5-32a7-4207-a0ab-0383e28b69ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:55.339196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:55.407131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 721/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:23:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:55.567091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19da614a-27cf-4cbb-8257-c699bffeb940 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:55.568627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-19da614a-27cf-4cbb-8257-c699bffeb940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:55.731058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19da614a-27cf-4cbb-8257-c699bffeb940 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:55.792742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:55.863540 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4a8637-6d2c-42b8-84d5-7da1a6fe16f1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:55.864158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4a8637-6d2c-42b8-84d5-7da1a6fe16f1 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:55.864431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4a8637-6d2c-42b8-84d5-7da1a6fe16f1 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:55.864709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4a8637-6d2c-42b8-84d5-7da1a6fe16f1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:55.866364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2e4a8637-6d2c-42b8-84d5-7da1a6fe16f1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:55.964942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4a8637-6d2c-42b8-84d5-7da1a6fe16f1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:55.966595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4a8637-6d2c-42b8-84d5-7da1a6fe16f1 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:55.986168 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 716/1437] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:57.038588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92b3977-0455-4c37-b11a-033275de6ac9 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:57.039226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92b3977-0455-4c37-b11a-033275de6ac9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:57.039226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92b3977-0455-4c37-b11a-033275de6ac9 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:57.040659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 722/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:23:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:57.048344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6c7c08-0073-4a62-bfd5-cb6997440328 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:57.049126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6c7c08-0073-4a62-bfd5-cb6997440328 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:57.049273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6c7c08-0073-4a62-bfd5-cb6997440328 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:57.139874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-9c6c7c08-0073-4a62-bfd5-cb6997440328 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:57.141094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9c6c7c08-0073-4a62-bfd5-cb6997440328 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:57.143353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:57.253460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 717/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:23:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:23:57.408681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92b3977-0455-4c37-b11a-033275de6ac9 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:23:57.410258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d92b3977-0455-4c37-b11a-033275de6ac9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:57.564847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92b3977-0455-4c37-b11a-033275de6ac9 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:23:57.592067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:57.687912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb644f4e-18b1-47f0-a40b-0b04ebb1a4d4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:23:57.688455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb644f4e-18b1-47f0-a40b-0b04ebb1a4d4 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:57.688733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb644f4e-18b1-47f0-a40b-0b04ebb1a4d4 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:57.689000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb644f4e-18b1-47f0-a40b-0b04ebb1a4d4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:57.690655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-eb644f4e-18b1-47f0-a40b-0b04ebb1a4d4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:23:57.785746 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb644f4e-18b1-47f0-a40b-0b04ebb1a4d4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:23:57.787368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb644f4e-18b1-47f0-a40b-0b04ebb1a4d4 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:23:57.810818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 723/1440] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:23:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:23:59.879175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56980cd8-9508-4be2-978c-8738004a7990 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:59.879742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56980cd8-9508-4be2-978c-8738004a7990 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:59.879742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56980cd8-9508-4be2-978c-8738004a7990 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:59.881134 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 718/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:23:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:23:59.887641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e710534e-071a-4fc4-9697-20e85106db0a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:23:59.888126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e710534e-071a-4fc4-9697-20e85106db0a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:23:59.888221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e710534e-071a-4fc4-9697-20e85106db0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:23:59.947581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:23:59.974354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e710534e-071a-4fc4-9697-20e85106db0a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:23:59.975569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e710534e-071a-4fc4-9697-20e85106db0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:00.086596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 724/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:23:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:00.369951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56980cd8-9508-4be2-978c-8738004a7990 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:00.371551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-56980cd8-9508-4be2-978c-8738004a7990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:00.483672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56980cd8-9508-4be2-978c-8738004a7990 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:00.520252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:00.565219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b628ec1-57c9-4da3-9754-95d261c3d078 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:00.565870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b628ec1-57c9-4da3-9754-95d261c3d078 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:00.566128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b628ec1-57c9-4da3-9754-95d261c3d078 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:00.566478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b628ec1-57c9-4da3-9754-95d261c3d078 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:00.568103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4b628ec1-57c9-4da3-9754-95d261c3d078 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:00.682630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b628ec1-57c9-4da3-9754-95d261c3d078 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:00.684069 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b628ec1-57c9-4da3-9754-95d261c3d078 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:00.702861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 719/1443] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:24:03.005472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966ce572-df52-4477-85fa-f974f5c93a28 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:03.006075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966ce572-df52-4477-85fa-f974f5c93a28 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:03.006075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966ce572-df52-4477-85fa-f974f5c93a28 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:03.008021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 725/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:24:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:03.014306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce2e43-82ba-492e-96a1-e1b8cd29d837 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:03.014782 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce2e43-82ba-492e-96a1-e1b8cd29d837 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:03.014886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce2e43-82ba-492e-96a1-e1b8cd29d837 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:03.081707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-a4ce2e43-82ba-492e-96a1-e1b8cd29d837 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:03.082930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a4ce2e43-82ba-492e-96a1-e1b8cd29d837 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:03.096497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:03.182298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 720/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:24:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:03.465485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966ce572-df52-4477-85fa-f974f5c93a28 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:03.467124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-966ce572-df52-4477-85fa-f974f5c93a28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:03.577125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966ce572-df52-4477-85fa-f974f5c93a28 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:03.592036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:03.660333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792bb242-351f-40bd-bde1-0e8cebd1821f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:03.660982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792bb242-351f-40bd-bde1-0e8cebd1821f service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:03.661257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792bb242-351f-40bd-bde1-0e8cebd1821f service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:03.661520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792bb242-351f-40bd-bde1-0e8cebd1821f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:03.663226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-792bb242-351f-40bd-bde1-0e8cebd1821f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:03.782047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792bb242-351f-40bd-bde1-0e8cebd1821f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:03.783438 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792bb242-351f-40bd-bde1-0e8cebd1821f service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:03.803163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 726/1446] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:24:06.345623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b96367-8d60-44f8-9e04-5c719218ab87 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:06.346236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b96367-8d60-44f8-9e04-5c719218ab87 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:06.346236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b96367-8d60-44f8-9e04-5c719218ab87 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:06.347550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 721/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:24:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:06.356030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35063224-a2f8-4c14-82bd-837a79f81bfb None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:06.356735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35063224-a2f8-4c14-82bd-837a79f81bfb None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:06.356888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35063224-a2f8-4c14-82bd-837a79f81bfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:06.411244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:06.429787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-35063224-a2f8-4c14-82bd-837a79f81bfb None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:06.431204 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-35063224-a2f8-4c14-82bd-837a79f81bfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:06.535320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 727/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:24:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:06.826616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b96367-8d60-44f8-9e04-5c719218ab87 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:06.828199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-76b96367-8d60-44f8-9e04-5c719218ab87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:06.943618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b96367-8d60-44f8-9e04-5c719218ab87 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:06.984256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:07.019515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15896cda-e51e-4e1f-843b-98823ebf1c92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:07.020121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15896cda-e51e-4e1f-843b-98823ebf1c92 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:07.020399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15896cda-e51e-4e1f-843b-98823ebf1c92 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:07.020656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15896cda-e51e-4e1f-843b-98823ebf1c92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:07.022298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-15896cda-e51e-4e1f-843b-98823ebf1c92 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:07.133045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15896cda-e51e-4e1f-843b-98823ebf1c92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:07.134560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15896cda-e51e-4e1f-843b-98823ebf1c92 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:07.152303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 722/1449] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:24:09.448472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b2a65a-43f8-4d32-a256-cbbdb9c18668 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:09.449102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b2a65a-43f8-4d32-a256-cbbdb9c18668 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:09.449102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b2a65a-43f8-4d32-a256-cbbdb9c18668 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:09.450407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 728/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:24:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:24:09.456057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2285008e-2324-4ce2-a6fc-7e3cad86bcfc None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:09.456520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2285008e-2324-4ce2-a6fc-7e3cad86bcfc None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:09.456629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2285008e-2324-4ce2-a6fc-7e3cad86bcfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:09.524667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2285008e-2324-4ce2-a6fc-7e3cad86bcfc None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:09.525879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2285008e-2324-4ce2-a6fc-7e3cad86bcfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:09.547817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:09.628300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 723/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:24:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:09.912347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b2a65a-43f8-4d32-a256-cbbdb9c18668 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:09.913924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b7b2a65a-43f8-4d32-a256-cbbdb9c18668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:10.027559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b2a65a-43f8-4d32-a256-cbbdb9c18668 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:10.050877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:10.116994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dae11d-eb9f-405e-be75-b0ceab086e81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:10.117619 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dae11d-eb9f-405e-be75-b0ceab086e81 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:10.117896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dae11d-eb9f-405e-be75-b0ceab086e81 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:10.118170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dae11d-eb9f-405e-be75-b0ceab086e81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:10.119916 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-61dae11d-eb9f-405e-be75-b0ceab086e81 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:10.236102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dae11d-eb9f-405e-be75-b0ceab086e81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:10.237441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dae11d-eb9f-405e-be75-b0ceab086e81 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:10.255837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 729/1452] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:24:12.515288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781bbd45-8b10-40cd-a704-4f8e86e902b2 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:12.515894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781bbd45-8b10-40cd-a704-4f8e86e902b2 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:12.515894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781bbd45-8b10-40cd-a704-4f8e86e902b2 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:12.517165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 724/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:24:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:12.523914 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2a3dcb-2106-47f4-b2dc-ffde84a79696 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:12.524585 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2a3dcb-2106-47f4-b2dc-ffde84a79696 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:12.524740 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2a3dcb-2106-47f4-b2dc-ffde84a79696 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:12.604480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:12.608761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-6e2a3dcb-2106-47f4-b2dc-ffde84a79696 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:12.611313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6e2a3dcb-2106-47f4-b2dc-ffde84a79696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:12.732024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 730/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:24:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:13.018253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781bbd45-8b10-40cd-a704-4f8e86e902b2 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:13.020709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-781bbd45-8b10-40cd-a704-4f8e86e902b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:13.139690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781bbd45-8b10-40cd-a704-4f8e86e902b2 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:13.203337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:13.229249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8b2a7e-064a-4cea-b2a7-1a094b9a114f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:13.229857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8b2a7e-064a-4cea-b2a7-1a094b9a114f service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:13.230199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8b2a7e-064a-4cea-b2a7-1a094b9a114f service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:13.230427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8b2a7e-064a-4cea-b2a7-1a094b9a114f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:13.232026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9f8b2a7e-064a-4cea-b2a7-1a094b9a114f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:13.350430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8b2a7e-064a-4cea-b2a7-1a094b9a114f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:13.352302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8b2a7e-064a-4cea-b2a7-1a094b9a114f service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:13.372262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 725/1455] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:24:15.609630 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456d37e1-dfbf-49ac-b0cd-dc5f37b47566 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:15.610252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456d37e1-dfbf-49ac-b0cd-dc5f37b47566 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:15.610252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456d37e1-dfbf-49ac-b0cd-dc5f37b47566 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:15.611563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 731/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:24:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:15.618250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231095f7-9db0-4c69-bfc8-e69d3c23fcef None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:15.618889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231095f7-9db0-4c69-bfc8-e69d3c23fcef None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:15.618889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231095f7-9db0-4c69-bfc8-e69d3c23fcef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:15.691676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-231095f7-9db0-4c69-bfc8-e69d3c23fcef None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:15.692903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-231095f7-9db0-4c69-bfc8-e69d3c23fcef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:15.724928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:15.795424 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 726/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:24:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:16.096509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456d37e1-dfbf-49ac-b0cd-dc5f37b47566 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:16.098085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-456d37e1-dfbf-49ac-b0cd-dc5f37b47566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:16.225448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456d37e1-dfbf-49ac-b0cd-dc5f37b47566 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:16.255836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:16.302522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fdb7c8-c9df-49a3-9ba2-d2cfd5a7e25b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:16.303136 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fdb7c8-c9df-49a3-9ba2-d2cfd5a7e25b service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:16.303408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fdb7c8-c9df-49a3-9ba2-d2cfd5a7e25b service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:16.303672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fdb7c8-c9df-49a3-9ba2-d2cfd5a7e25b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:16.305274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-06fdb7c8-c9df-49a3-9ba2-d2cfd5a7e25b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:16.429050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fdb7c8-c9df-49a3-9ba2-d2cfd5a7e25b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:16.430522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fdb7c8-c9df-49a3-9ba2-d2cfd5a7e25b service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:16.449248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 732/1458] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:24:18.637054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15dd44b-7a9b-4668-9389-133bfc5ede20 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:18.637632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15dd44b-7a9b-4668-9389-133bfc5ede20 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:18.637632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15dd44b-7a9b-4668-9389-133bfc5ede20 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:18.638952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 727/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:24:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:18.645003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a87698-a007-47f1-be0e-51dd7e6fc460 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:18.645488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a87698-a007-47f1-be0e-51dd7e6fc460 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:18.645602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a87698-a007-47f1-be0e-51dd7e6fc460 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:18.717170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-88a87698-a007-47f1-be0e-51dd7e6fc460 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:18.718470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-88a87698-a007-47f1-be0e-51dd7e6fc460 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:18.727554 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:18.820658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 733/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:24:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:24:19.125674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15dd44b-7a9b-4668-9389-133bfc5ede20 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:19.127298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a15dd44b-7a9b-4668-9389-133bfc5ede20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:19.245296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15dd44b-7a9b-4668-9389-133bfc5ede20 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:19.305347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:19.324813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c91e0a-2eb4-4687-be83-75073c56ddf7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:19.325430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c91e0a-2eb4-4687-be83-75073c56ddf7 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:19.325702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c91e0a-2eb4-4687-be83-75073c56ddf7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:19.325975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c91e0a-2eb4-4687-be83-75073c56ddf7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:19.327612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-96c91e0a-2eb4-4687-be83-75073c56ddf7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:19.443919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c91e0a-2eb4-4687-be83-75073c56ddf7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:19.445384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c91e0a-2eb4-4687-be83-75073c56ddf7 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:19.464047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 728/1461] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:24:21.668441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e422393f-c55f-4218-a7ea-4f628d9ba8be None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:21.668964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e422393f-c55f-4218-a7ea-4f628d9ba8be None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:21.668964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e422393f-c55f-4218-a7ea-4f628d9ba8be None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:21.670332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 734/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:24:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:21.675921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b857886-23a7-42c4-916c-5395d2828d0f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:21.676341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b857886-23a7-42c4-916c-5395d2828d0f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:21.676467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b857886-23a7-42c4-916c-5395d2828d0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:21.753126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-4b857886-23a7-42c4-916c-5395d2828d0f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:21.754409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4b857886-23a7-42c4-916c-5395d2828d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:21.795891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:21.862001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 729/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:24:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:22.148230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e422393f-c55f-4218-a7ea-4f628d9ba8be None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:22.149904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e422393f-c55f-4218-a7ea-4f628d9ba8be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:22.269022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e422393f-c55f-4218-a7ea-4f628d9ba8be None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:22.309823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:22.347352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b558444-cb3a-4c87-8430-043b1b8e48d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:22.347973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b558444-cb3a-4c87-8430-043b1b8e48d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:22.348246 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b558444-cb3a-4c87-8430-043b1b8e48d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:22.348517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b558444-cb3a-4c87-8430-043b1b8e48d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:22.350528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3b558444-cb3a-4c87-8430-043b1b8e48d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:22.477299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b558444-cb3a-4c87-8430-043b1b8e48d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:22.478899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b558444-cb3a-4c87-8430-043b1b8e48d0 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:22.498030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 735/1464] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:24:24.683507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2143f56-45f5-46f6-b5af-f523832840bf None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:24.684380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2143f56-45f5-46f6-b5af-f523832840bf None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:24.684380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2143f56-45f5-46f6-b5af-f523832840bf None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:24.686521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 730/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:24:24 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:24.692001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58e8040-05d8-45d3-9454-fa0e03580696 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:24.692448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58e8040-05d8-45d3-9454-fa0e03580696 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:24.692558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58e8040-05d8-45d3-9454-fa0e03580696 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:24.770851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e58e8040-05d8-45d3-9454-fa0e03580696 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:24.772107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e58e8040-05d8-45d3-9454-fa0e03580696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:24.791539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:24.878396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 736/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:24:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:25.166350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2143f56-45f5-46f6-b5af-f523832840bf None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:25.167948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d2143f56-45f5-46f6-b5af-f523832840bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:25.280141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2143f56-45f5-46f6-b5af-f523832840bf None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:25.355529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:25.362914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e6e432-77d9-471b-9ed0-ad0f5fa5711c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:25.363532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e6e432-77d9-471b-9ed0-ad0f5fa5711c service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:25.363810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e6e432-77d9-471b-9ed0-ad0f5fa5711c service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:25.364109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e6e432-77d9-471b-9ed0-ad0f5fa5711c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:25.365799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-40e6e432-77d9-471b-9ed0-ad0f5fa5711c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:25.486015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e6e432-77d9-471b-9ed0-ad0f5fa5711c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:25.487540 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e6e432-77d9-471b-9ed0-ad0f5fa5711c service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:25.506289 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 731/1467] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:24:27.670320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3f1b5b-3d54-462d-83d1-0b2df30b1580 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:27.670846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3f1b5b-3d54-462d-83d1-0b2df30b1580 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:27.670962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3f1b5b-3d54-462d-83d1-0b2df30b1580 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:27.672950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 737/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:24:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:27.680627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96acb0c8-c8f2-4aa5-a8ae-e3cd74cd0cce None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:27.681049 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96acb0c8-c8f2-4aa5-a8ae-e3cd74cd0cce None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:27.681208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96acb0c8-c8f2-4aa5-a8ae-e3cd74cd0cce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:27.752763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-96acb0c8-c8f2-4aa5-a8ae-e3cd74cd0cce None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:27.753987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-96acb0c8-c8f2-4aa5-a8ae-e3cd74cd0cce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:27.802306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:27.869684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 732/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:24:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:28.168232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3f1b5b-3d54-462d-83d1-0b2df30b1580 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:28.170716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9b3f1b5b-3d54-462d-83d1-0b2df30b1580 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:28.291433 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3f1b5b-3d54-462d-83d1-0b2df30b1580 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:28.355221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:28.382658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667f208a-87aa-4b9a-9359-2a7ef5a92701 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:28.383276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667f208a-87aa-4b9a-9359-2a7ef5a92701 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:28.383556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667f208a-87aa-4b9a-9359-2a7ef5a92701 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:28.383826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667f208a-87aa-4b9a-9359-2a7ef5a92701 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:28.385716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-667f208a-87aa-4b9a-9359-2a7ef5a92701 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:28.511622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667f208a-87aa-4b9a-9359-2a7ef5a92701 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:28.512969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667f208a-87aa-4b9a-9359-2a7ef5a92701 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:28.535014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 738/1470] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:24:37.867020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=22 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:38.531148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=66 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:48.765906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e02827-d287-4ed3-b743-3ba00e6dd11f None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:48.766548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e02827-d287-4ed3-b743-3ba00e6dd11f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:48.766548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e02827-d287-4ed3-b743-3ba00e6dd11f None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:48.767855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 733/1471] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:24:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:48.774474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5ccc1-2daf-44ea-96ed-c5dbca3cc4a3 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:48.775224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5ccc1-2daf-44ea-96ed-c5dbca3cc4a3 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:48.775649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5ccc1-2daf-44ea-96ed-c5dbca3cc4a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:48.851922 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-bac5ccc1-2daf-44ea-96ed-c5dbca3cc4a3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:48.853049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-bac5ccc1-2daf-44ea-96ed-c5dbca3cc4a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:48.966528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 739/1472] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:24:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:49.276365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94ee27c-dcb9-48fc-9372-96bebdce28fe None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:49.276642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94ee27c-dcb9-48fc-9372-96bebdce28fe None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:49.276902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94ee27c-dcb9-48fc-9372-96bebdce28fe None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:49.278311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 734/1473] 10.222.0.16 () {54 vars in 966 bytes} [Tue Sep 13 15:24:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:49.286320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4bf25-6e4a-4556-a122-5dcd766f874e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:49.286735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4bf25-6e4a-4556-a122-5dcd766f874e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:49.287131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4bf25-6e4a-4556-a122-5dcd766f874e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:49.368346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-eca4bf25-6e4a-4556-a122-5dcd766f874e None None] MFA Rules not processed for user `2ce646dcb8124685ac7a1c1eca664d04`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:49.369745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-eca4bf25-6e4a-4556-a122-5dcd766f874e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:49.390376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:49.446854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 740/1474] 10.222.0.16 () {58 vars in 1063 bytes} [Tue Sep 13 15:24:49 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:49.454975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94ee27c-dcb9-48fc-9372-96bebdce28fe None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:49.456493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a94ee27c-dcb9-48fc-9372-96bebdce28fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:49.564189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94ee27c-dcb9-48fc-9372-96bebdce28fe None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:49.639908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01bcb6e-67f0-4f35-baf8-4d743d50b6e2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce646dcb8124685ac7a1c1eca664d04', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:49.640514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01bcb6e-67f0-4f35-baf8-4d743d50b6e2 service glance] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:49.640793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01bcb6e-67f0-4f35-baf8-4d743d50b6e2 service glance] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:49.641061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01bcb6e-67f0-4f35-baf8-4d743d50b6e2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:49.642676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a01bcb6e-67f0-4f35-baf8-4d743d50b6e2 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:49.698875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:49.758007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01bcb6e-67f0-4f35-baf8-4d743d50b6e2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:49.759553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01bcb6e-67f0-4f35-baf8-4d743d50b6e2 service glance] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:49.778957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 735/1475] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:24:52.013416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6ba013-76ad-43b8-9be9-c88bf78e652d None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:52.014363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6ba013-76ad-43b8-9be9-c88bf78e652d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:52.014363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6ba013-76ad-43b8-9be9-c88bf78e652d None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:52.016523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 741/1476] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:24:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:52.024857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e11dd7-fbff-432b-a16f-7e8bbd10a1d8 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:52.025434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e11dd7-fbff-432b-a16f-7e8bbd10a1d8 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:52.025515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e11dd7-fbff-432b-a16f-7e8bbd10a1d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:52.094351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-27e11dd7-fbff-432b-a16f-7e8bbd10a1d8 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:52.095569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-27e11dd7-fbff-432b-a16f-7e8bbd10a1d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:52.204228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 736/1477] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:24:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:24:52.473088 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6ba013-76ad-43b8-9be9-c88bf78e652d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:52.474691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-be6ba013-76ad-43b8-9be9-c88bf78e652d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:52.615284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6ba013-76ad-43b8-9be9-c88bf78e652d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:52.669908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:52.724164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f2442f-60fb-42fb-b463-b8d74a148fca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:52.724699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f2442f-60fb-42fb-b463-b8d74a148fca admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:52.724974 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f2442f-60fb-42fb-b463-b8d74a148fca admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:52.725241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f2442f-60fb-42fb-b463-b8d74a148fca admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:52.726264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f2442f-60fb-42fb-b463-b8d74a148fca admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:52.728068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: WARNING py.warnings [None req-a2f2442f-60fb-42fb-b463-b8d74a148fca admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:24:52.728068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: warnings.warn(msg) Sep 13 15:24:52.728068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: Sep 13 15:24:52.729835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f2442f-60fb-42fb-b463-b8d74a148fca admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:52.752139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 742/1478] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Sep 13 15:24:52 2022] GET /identity/v3/projects => generated 2553 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:24:54.915146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9d2615-d10b-42fd-a464-9ab5d60c21f3 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:54.915878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9d2615-d10b-42fd-a464-9ab5d60c21f3 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:54.916091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9d2615-d10b-42fd-a464-9ab5d60c21f3 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:54.918501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 737/1479] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:24:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:54.924220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b546894-a139-4a82-b0dd-609dcbe7d71d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:54.924688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b546894-a139-4a82-b0dd-609dcbe7d71d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:54.924801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b546894-a139-4a82-b0dd-609dcbe7d71d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:54.992812 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-5b546894-a139-4a82-b0dd-609dcbe7d71d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:54.994090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5b546894-a139-4a82-b0dd-609dcbe7d71d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:55.095638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 743/1480] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:24:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:55.391923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9d2615-d10b-42fd-a464-9ab5d60c21f3 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:55.393571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0e9d2615-d10b-42fd-a464-9ab5d60c21f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:55.403977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:55.547053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9d2615-d10b-42fd-a464-9ab5d60c21f3 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:55.637426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:55.670898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6923e5fe-891e-4321-92c5-bb0ffa6d7a3d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:55.671511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6923e5fe-891e-4321-92c5-bb0ffa6d7a3d service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:55.671787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6923e5fe-891e-4321-92c5-bb0ffa6d7a3d service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:55.672052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6923e5fe-891e-4321-92c5-bb0ffa6d7a3d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:55.673651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6923e5fe-891e-4321-92c5-bb0ffa6d7a3d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:55.767002 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6923e5fe-891e-4321-92c5-bb0ffa6d7a3d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:55.768612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6923e5fe-891e-4321-92c5-bb0ffa6d7a3d service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:55.787781 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 738/1481] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:24:58.139931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb68624-316b-4a3c-8021-3f56135393c5 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:58.140904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb68624-316b-4a3c-8021-3f56135393c5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:58.140904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb68624-316b-4a3c-8021-3f56135393c5 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:58.143013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 744/1482] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:24:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:24:58.153864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9fccc1-e030-46bc-887f-bd91bd7026f0 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:58.154343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9fccc1-e030-46bc-887f-bd91bd7026f0 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:58.154705 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9fccc1-e030-46bc-887f-bd91bd7026f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:58.233543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-5b9fccc1-e030-46bc-887f-bd91bd7026f0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:24:58.234819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-5b9fccc1-e030-46bc-887f-bd91bd7026f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:58.297219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:58.353044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 739/1483] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:24:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:24:58.644311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb68624-316b-4a3c-8021-3f56135393c5 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:24:58.645925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7eb68624-316b-4a3c-8021-3f56135393c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:58.671218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:58.768817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb68624-316b-4a3c-8021-3f56135393c5 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:24:58.846774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea560f54-3837-4172-890f-435be484aca8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:24:58.847384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea560f54-3837-4172-890f-435be484aca8 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:24:58.847641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea560f54-3837-4172-890f-435be484aca8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:24:58.847907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea560f54-3837-4172-890f-435be484aca8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:24:58.849505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ea560f54-3837-4172-890f-435be484aca8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:24:58.932480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:24:58.969455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea560f54-3837-4172-890f-435be484aca8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:24:58.970944 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea560f54-3837-4172-890f-435be484aca8 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:24:58.990510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 745/1484] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:24:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:25:01.190986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4fdf6b-05ca-47ab-89b6-a46fcd48e005 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:25:01.191859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4fdf6b-05ca-47ab-89b6-a46fcd48e005 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:25:01.191859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4fdf6b-05ca-47ab-89b6-a46fcd48e005 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:25:01.193992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 740/1485] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:25:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:25:01.200720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2026ddee-464a-439b-9765-f5da7a501860 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:25:01.201178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2026ddee-464a-439b-9765-f5da7a501860 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:25:01.201407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2026ddee-464a-439b-9765-f5da7a501860 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:25:01.278367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-2026ddee-464a-439b-9765-f5da7a501860 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:25:01.279584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2026ddee-464a-439b-9765-f5da7a501860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:25:01.381376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 746/1486] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:25:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:25:01.679637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4fdf6b-05ca-47ab-89b6-a46fcd48e005 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:25:01.682310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9f4fdf6b-05ca-47ab-89b6-a46fcd48e005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:25:01.796288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4fdf6b-05ca-47ab-89b6-a46fcd48e005 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:25:01.885198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:25:01.886496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b6fd4d-879d-4ed6-bc0f-1938bb54ca43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:25:01.887110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b6fd4d-879d-4ed6-bc0f-1938bb54ca43 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:25:01.887415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b6fd4d-879d-4ed6-bc0f-1938bb54ca43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:25:01.887722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b6fd4d-879d-4ed6-bc0f-1938bb54ca43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:25:01.890124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-01b6fd4d-879d-4ed6-bc0f-1938bb54ca43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:25:02.017815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b6fd4d-879d-4ed6-bc0f-1938bb54ca43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:25:02.019311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b6fd4d-879d-4ed6-bc0f-1938bb54ca43 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:25:02.038326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 741/1487] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:25:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:25:04.349246 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93ceaf8-ee74-4018-9461-062b1680d75e None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:25:04.350190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93ceaf8-ee74-4018-9461-062b1680d75e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:25:04.350190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93ceaf8-ee74-4018-9461-062b1680d75e None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:25:04.352687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 747/1488] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:25:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:25:04.358794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d76209-56d4-404c-ab34-7da9be1267ec None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:25:04.359296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d76209-56d4-404c-ab34-7da9be1267ec None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:25:04.359396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d76209-56d4-404c-ab34-7da9be1267ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:25:04.434942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-56d76209-56d4-404c-ab34-7da9be1267ec None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:25:04.436150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-56d76209-56d4-404c-ab34-7da9be1267ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:25:04.493013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:25:04.546898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 742/1489] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:25:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:25:04.834883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93ceaf8-ee74-4018-9461-062b1680d75e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:25:04.837477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a93ceaf8-ee74-4018-9461-062b1680d75e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:25:04.875690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:25:04.945712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93ceaf8-ee74-4018-9461-062b1680d75e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:25:05.022483 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6f5392-a319-4ae9-8b07-1e79ec7ebb98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:25:05.023085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6f5392-a319-4ae9-8b07-1e79ec7ebb98 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:25:05.023367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6f5392-a319-4ae9-8b07-1e79ec7ebb98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:25:05.023642 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6f5392-a319-4ae9-8b07-1e79ec7ebb98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:25:05.025273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9e6f5392-a319-4ae9-8b07-1e79ec7ebb98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:25:05.113724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:25:05.147897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6f5392-a319-4ae9-8b07-1e79ec7ebb98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:25:05.149330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6f5392-a319-4ae9-8b07-1e79ec7ebb98 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:25:05.167681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 748/1490] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:25:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:25:07.588355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353ff8f6-f9f8-47dd-add2-fcb144e16ed0 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:25:07.589128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353ff8f6-f9f8-47dd-add2-fcb144e16ed0 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:25:07.589128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353ff8f6-f9f8-47dd-add2-fcb144e16ed0 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:25:07.590272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 743/1491] 10.222.0.16 () {54 vars in 927 bytes} [Tue Sep 13 15:25:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:25:07.599175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1bfb2-426d-4676-b40b-ba2ab1ba321a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:25:07.599876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1bfb2-426d-4676-b40b-ba2ab1ba321a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:25:07.600046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee1bfb2-426d-4676-b40b-ba2ab1ba321a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:25:07.678087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8ee1bfb2-426d-4676-b40b-ba2ab1ba321a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:25:07.679329 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8ee1bfb2-426d-4676-b40b-ba2ab1ba321a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:25:07.780282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 749/1492] 10.222.0.16 () {58 vars in 1024 bytes} [Tue Sep 13 15:25:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:25:08.071001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353ff8f6-f9f8-47dd-add2-fcb144e16ed0 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:25:08.073207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-353ff8f6-f9f8-47dd-add2-fcb144e16ed0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:25:08.198563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353ff8f6-f9f8-47dd-add2-fcb144e16ed0 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:25:08.275627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e766709e-6c23-4c5f-bf5c-a92644f070b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:25:08.276237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e766709e-6c23-4c5f-bf5c-a92644f070b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:25:08.276513 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e766709e-6c23-4c5f-bf5c-a92644f070b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:25:08.276783 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e766709e-6c23-4c5f-bf5c-a92644f070b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:25:08.278404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e766709e-6c23-4c5f-bf5c-a92644f070b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:25:08.287730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:25:08.400066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e766709e-6c23-4c5f-bf5c-a92644f070b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:25:08.401627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e766709e-6c23-4c5f-bf5c-a92644f070b0 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:25:08.419089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 744/1493] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:25:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:25:17.778760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=82 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:25:18.417116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=50 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:22.108367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512af7b4-04f4-4641-a33d-aa4148e76a4e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:22.109396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512af7b4-04f4-4641-a33d-aa4148e76a4e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:22.109396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512af7b4-04f4-4641-a33d-aa4148e76a4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:22.190954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-512af7b4-04f4-4641-a33d-aa4148e76a4e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:26:22.192160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-512af7b4-04f4-4641-a33d-aa4148e76a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:22.297684 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 750/1494] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:26:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:26:22.307077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912f140-30da-49ab-959c-1c38b8ddb6b3 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:22.307602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912f140-30da-49ab-959c-1c38b8ddb6b3 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:22.307602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912f140-30da-49ab-959c-1c38b8ddb6b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:22.399808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3912f140-30da-49ab-959c-1c38b8ddb6b3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:26:22.401045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3912f140-30da-49ab-959c-1c38b8ddb6b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:22.505044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 745/1495] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:26:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:26:22.510566 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512af7b4-04f4-4641-a33d-aa4148e76a4e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:26:22.512147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-512af7b4-04f4-4641-a33d-aa4148e76a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:22.523912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512af7b4-04f4-4641-a33d-aa4148e76a4e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:26:22.603534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:22.642451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:26:22.643044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:22.643314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:22.643579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:22.644224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:26:22.644521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:26:22.645534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: WARNING py.warnings [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:26:22.645534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: warnings.warn(msg) Sep 13 15:26:22.645534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: Sep 13 15:26:22.646460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:26:22.659976 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 751/1496] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:26:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:26:22.668845 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3912f140-30da-49ab-959c-1c38b8ddb6b3 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:26:22.678404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3912f140-30da-49ab-959c-1c38b8ddb6b3 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:26:22.746330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:22.779948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d07e0c3-6abb-4c90-9198-7c121c2540f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:26:22.780560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d07e0c3-6abb-4c90-9198-7c121c2540f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:22.780836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d07e0c3-6abb-4c90-9198-7c121c2540f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:22.781100 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d07e0c3-6abb-4c90-9198-7c121c2540f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:22.781724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d07e0c3-6abb-4c90-9198-7c121c2540f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:26:22.783506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: WARNING py.warnings [None req-2d07e0c3-6abb-4c90-9198-7c121c2540f9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:26:22.783506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: warnings.warn(msg) Sep 13 15:26:22.783506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: Sep 13 15:26:22.784689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d07e0c3-6abb-4c90-9198-7c121c2540f9 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:26:22.850305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 746/1497] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:26:22 2022] POST /identity/v3/projects => generated 377 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:26:22.855337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:26:22.856863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:22.889843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501f3853-4088-4c4a-8a2c-2b3112f5b3c0 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:26:23.000935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf80b228-dade-4430-8809-a582bfc14d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:26:23.001542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf80b228-dade-4430-8809-a582bfc14d2e admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:23.001816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf80b228-dade-4430-8809-a582bfc14d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:23.002083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf80b228-dade-4430-8809-a582bfc14d2e admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:23.002733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf80b228-dade-4430-8809-a582bfc14d2e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:26:23.003754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: WARNING py.warnings [None req-cf80b228-dade-4430-8809-a582bfc14d2e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:26:23.003754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: warnings.warn(msg) Sep 13 15:26:23.003754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: Sep 13 15:26:23.004948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf80b228-dade-4430-8809-a582bfc14d2e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:26:23.052475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 752/1498] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:26:22 2022] POST /identity/v3/users => generated 289 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:26:23.056905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d07e0c3-6abb-4c90-9198-7c121c2540f9 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:26:23.058501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2d07e0c3-6abb-4c90-9198-7c121c2540f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:23.132598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d07e0c3-6abb-4c90-9198-7c121c2540f9 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:26:23.227676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:23.239884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239091be-0f4b-4638-9ba9-d053cb750994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:26:23.240493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239091be-0f4b-4638-9ba9-d053cb750994 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:23.240765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239091be-0f4b-4638-9ba9-d053cb750994 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:23.241025 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239091be-0f4b-4638-9ba9-d053cb750994 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:23.241658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239091be-0f4b-4638-9ba9-d053cb750994 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:26:23.242714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: WARNING py.warnings [None req-239091be-0f4b-4638-9ba9-d053cb750994 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:26:23.242714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: warnings.warn(msg) Sep 13 15:26:23.242714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: Sep 13 15:26:23.243592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239091be-0f4b-4638-9ba9-d053cb750994 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:26:23.256189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 747/1499] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:26:23 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:26:23.260270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf80b228-dade-4430-8809-a582bfc14d2e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:26:23.261891 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-cf80b228-dade-4430-8809-a582bfc14d2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:23.273330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf80b228-dade-4430-8809-a582bfc14d2e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:26:23.300302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:23.372591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8634c6-4380-4d91-a246-f128465ee936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:26:23.373297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8634c6-4380-4d91-a246-f128465ee936 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:23.373594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8634c6-4380-4d91-a246-f128465ee936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:23.373924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8634c6-4380-4d91-a246-f128465ee936 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:23.374595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8634c6-4380-4d91-a246-f128465ee936 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:26:23.375611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: WARNING py.warnings [None req-fb8634c6-4380-4d91-a246-f128465ee936 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:26:23.375611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: warnings.warn(msg) Sep 13 15:26:23.375611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: Sep 13 15:26:23.376484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8634c6-4380-4d91-a246-f128465ee936 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:26:23.393768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 753/1500] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:26:23 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:26:23.398356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239091be-0f4b-4638-9ba9-d053cb750994 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:26:23.399931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-239091be-0f4b-4638-9ba9-d053cb750994 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:23.435145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239091be-0f4b-4638-9ba9-d053cb750994 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:26:23.551242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896080db-3e0d-48ae-8ebc-e734c79ba9a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:26:23.551821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896080db-3e0d-48ae-8ebc-e734c79ba9a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:23.552094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896080db-3e0d-48ae-8ebc-e734c79ba9a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:23.552366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896080db-3e0d-48ae-8ebc-e734c79ba9a9 admin admin] PATH_INFO: `/v3/projects/2f5c0538fabe4cdbae7076993ee3ec00/users/f0db5479d6174b92ac0ef6a9f8e6635b/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:23.577092 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896080db-3e0d-48ae-8ebc-e734c79ba9a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f5c0538fabe4cdbae7076993ee3ec00, user_id=f0db5479d6174b92ac0ef6a9f8e6635b, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:26:23.578146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: WARNING py.warnings [None req-896080db-3e0d-48ae-8ebc-e734c79ba9a9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:26:23.578146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: warnings.warn(msg) Sep 13 15:26:23.578146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: Sep 13 15:26:23.580133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896080db-3e0d-48ae-8ebc-e734c79ba9a9 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:26:23.608514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 748/1501] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:26:23 2022] PUT /identity/v3/projects/2f5c0538fabe4cdbae7076993ee3ec00/users/f0db5479d6174b92ac0ef6a9f8e6635b/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:26:23.621153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f570de2e-48cc-48fa-9030-c1d17e9b965f None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:23.621459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f570de2e-48cc-48fa-9030-c1d17e9b965f None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:23.621757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f570de2e-48cc-48fa-9030-c1d17e9b965f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:23.686176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-f570de2e-48cc-48fa-9030-c1d17e9b965f None None] MFA Rules not processed for user `f0db5479d6174b92ac0ef6a9f8e6635b`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:26:23.687435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f570de2e-48cc-48fa-9030-c1d17e9b965f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:23.769524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 754/1502] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:26:23 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:26:23.783645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0554b3-0767-4010-9ebe-dc790fa963b3 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:23.783927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0554b3-0767-4010-9ebe-dc790fa963b3 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:23.784201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0554b3-0767-4010-9ebe-dc790fa963b3 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:23.785552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 749/1503] 10.222.0.16 () {54 vars in 964 bytes} [Tue Sep 13 15:26:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:26:23.790777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ceb4a-5141-43af-a663-f87b857e1d03 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:23.791060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ceb4a-5141-43af-a663-f87b857e1d03 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:23.791329 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ceb4a-5141-43af-a663-f87b857e1d03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:23.825032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:23.874926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d06ceb4a-5141-43af-a663-f87b857e1d03 None None] MFA Rules not processed for user `39c54901c5b5400f96e084fd4adf54e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:26:23.876146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d06ceb4a-5141-43af-a663-f87b857e1d03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:23.957108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 755/1504] 10.222.0.16 () {58 vars in 1061 bytes} [Tue Sep 13 15:26:23 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:26:23.966020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0554b3-0767-4010-9ebe-dc790fa963b3 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:26:23.967609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6a0554b3-0767-4010-9ebe-dc790fa963b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:24.023969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:24.079313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0554b3-0767-4010-9ebe-dc790fa963b3 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:26:24.158472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479ad9fd-9448-4062-89db-cc11c349775e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:26:24.159085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479ad9fd-9448-4062-89db-cc11c349775e service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:24.159362 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479ad9fd-9448-4062-89db-cc11c349775e service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:24.159632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479ad9fd-9448-4062-89db-cc11c349775e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:24.161251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-479ad9fd-9448-4062-89db-cc11c349775e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:24.265524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479ad9fd-9448-4062-89db-cc11c349775e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:26:24.267027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479ad9fd-9448-4062-89db-cc11c349775e service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:26:24.270436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:24.286345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 750/1505] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:26:23 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:26:24.342486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1ccb5c-de36-4996-b52e-3ad943dfdc82 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:24.342881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1ccb5c-de36-4996-b52e-3ad943dfdc82 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:24.343026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1ccb5c-de36-4996-b52e-3ad943dfdc82 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:24.344320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 756/1506] 10.222.0.16 () {52 vars in 830 bytes} [Tue Sep 13 15:26:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:26:24.348231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479ad9fd-9448-4062-89db-cc11c349775e service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:26:24.349489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-479ad9fd-9448-4062-89db-cc11c349775e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:24.460457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479ad9fd-9448-4062-89db-cc11c349775e service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:26:24.564659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:26:24.565305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:24.565579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:24.565860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] PATH_INFO: `/v3/users/f0db5479d6174b92ac0ef6a9f8e6635b` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:24.579930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0db5479d6174b92ac0ef6a9f8e6635b)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:26:24.580979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: WARNING py.warnings [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:26:24.580979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: warnings.warn(msg) Sep 13 15:26:24.580979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: Sep 13 15:26:24.582241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:26:24.592745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:24.656094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0db5479d6174b92ac0ef6a9f8e6635b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:26:24.656641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0db5479d6174b92ac0ef6a9f8e6635b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:26:24.670476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0db5479d6174b92ac0ef6a9f8e6635b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:26:24.670910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0db5479d6174b92ac0ef6a9f8e6635b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:26:24.682739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0db5479d6174b92ac0ef6a9f8e6635b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:26:24.711939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 751/1507] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:26:24 2022] DELETE /identity/v3/users/f0db5479d6174b92ac0ef6a9f8e6635b => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:26:24.716795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a5b012-5aab-41f1-b3d2-35e699a5affb None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:24.717094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a5b012-5aab-41f1-b3d2-35e699a5affb None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:24.717365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a5b012-5aab-41f1-b3d2-35e699a5affb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:24.776974 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b3a5b012-5aab-41f1-b3d2-35e699a5affb None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:26:24.778400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b3a5b012-5aab-41f1-b3d2-35e699a5affb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:24.855877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:24.869093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 757/1508] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:26:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:26:24.879716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:26:24.880938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:24.981827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ddc62f-e3d8-46b8-9a87-0fb3eb8f0443 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:26:25.039527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:25.060114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b317eb2d-705c-4a51-a02a-f95567b7a6c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:26:25.060731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b317eb2d-705c-4a51-a02a-f95567b7a6c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:25.061011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b317eb2d-705c-4a51-a02a-f95567b7a6c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:25.061284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b317eb2d-705c-4a51-a02a-f95567b7a6c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:25.062935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b317eb2d-705c-4a51-a02a-f95567b7a6c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:25.189519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b317eb2d-705c-4a51-a02a-f95567b7a6c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:26:25.191083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b317eb2d-705c-4a51-a02a-f95567b7a6c7 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:26:25.213034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 752/1509] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:26:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:26:25.579527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a5b012-5aab-41f1-b3d2-35e699a5affb None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:26:25.580770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b3a5b012-5aab-41f1-b3d2-35e699a5affb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:26:25.594170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a5b012-5aab-41f1-b3d2-35e699a5affb None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:26:25.697461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:26:25.698013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:26:25.698321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:26:25.698589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] PATH_INFO: `/v3/projects/2f5c0538fabe4cdbae7076993ee3ec00` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:26:25.710664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] RBAC: Authorizing `identity:delete_project(project_id=2f5c0538fabe4cdbae7076993ee3ec00)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:26:25.711696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: WARNING py.warnings [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:26:25.711696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: warnings.warn(msg) Sep 13 15:26:25.711696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: Sep 13 15:26:25.712883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:26:25.789275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2f5c0538fabe4cdbae7076993ee3ec00', 'request_id': 'req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:26:25.797120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2f5c0538fabe4cdbae7076993ee3ec00', 'request_id': 'req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:26:25.809311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2f5c0538fabe4cdbae7076993ee3ec00', 'request_id': 'req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:26:25.842240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] The token cache is being invalidate because project 2f5c0538fabe4cdbae7076993ee3ec00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:26:25.842520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:26:25.842942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e79c1244-eeb2-4b5f-b8fe-c31aaf7a9f2e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:26:25.844366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 758/1510] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:26:25 2022] DELETE /identity/v3/projects/2f5c0538fabe4cdbae7076993ee3ec00 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:26:35.208933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=62 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:26:35.839655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:27:55.169979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef30f07-5971-45ae-b7f9-7866778553fe None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:27:55.170656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef30f07-5971-45ae-b7f9-7866778553fe None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:27:55.170656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef30f07-5971-45ae-b7f9-7866778553fe None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:27:55.172702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 753/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:27:55 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:27:55.180732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b8d1d6-63b8-4e8f-8adf-8ba769441605 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:27:55.181198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b8d1d6-63b8-4e8f-8adf-8ba769441605 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:27:55.181302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b8d1d6-63b8-4e8f-8adf-8ba769441605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:27:55.276673 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-63b8d1d6-63b8-4e8f-8adf-8ba769441605 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:27:55.277804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-63b8d1d6-63b8-4e8f-8adf-8ba769441605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:27:55.389697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 759/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:27:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:27:55.546775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef30f07-5971-45ae-b7f9-7866778553fe None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:27:55.549255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-cef30f07-5971-45ae-b7f9-7866778553fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:27:55.675771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef30f07-5971-45ae-b7f9-7866778553fe None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:27:55.681741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf7c3ef-1df8-428e-83c1-c0955dc34702 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:27:55.682392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf7c3ef-1df8-428e-83c1-c0955dc34702 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:27:55.682667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf7c3ef-1df8-428e-83c1-c0955dc34702 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:27:55.682935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf7c3ef-1df8-428e-83c1-c0955dc34702 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:27:55.691848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf7c3ef-1df8-428e-83c1-c0955dc34702 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:27:55.693450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf7c3ef-1df8-428e-83c1-c0955dc34702 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:27:55.703393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 754/1513] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:27:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:27:56.787532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be02ee16-5dab-4569-8221-d8de0dbbdf4b None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:27:56.788077 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be02ee16-5dab-4569-8221-d8de0dbbdf4b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:27:56.788077 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be02ee16-5dab-4569-8221-d8de0dbbdf4b None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:27:56.789287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 760/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:27:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:27:56.795863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de33f9-cb30-450c-bfe8-5f683f6071c7 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:27:56.796385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de33f9-cb30-450c-bfe8-5f683f6071c7 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:27:56.796385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4de33f9-cb30-450c-bfe8-5f683f6071c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:27:56.818236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e4de33f9-cb30-450c-bfe8-5f683f6071c7 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:27:56.819522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e4de33f9-cb30-450c-bfe8-5f683f6071c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:27:56.827937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 755/1515] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:27:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:27:56.979448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be02ee16-5dab-4569-8221-d8de0dbbdf4b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:27:56.996422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be02ee16-5dab-4569-8221-d8de0dbbdf4b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:27:57.003083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726e42ad-3164-4ae7-ba01-8eaf2a12e87a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:27:57.003696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726e42ad-3164-4ae7-ba01-8eaf2a12e87a service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:27:57.003946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726e42ad-3164-4ae7-ba01-8eaf2a12e87a service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:27:57.004232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726e42ad-3164-4ae7-ba01-8eaf2a12e87a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:27:57.012801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726e42ad-3164-4ae7-ba01-8eaf2a12e87a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:27:57.014410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726e42ad-3164-4ae7-ba01-8eaf2a12e87a service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:27:57.024361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 761/1516] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:27:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:27:57.500098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cc015d-f0db-4ef7-90bc-ff57e650ef9d None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:27:57.500382 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cc015d-f0db-4ef7-90bc-ff57e650ef9d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:27:57.500628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cc015d-f0db-4ef7-90bc-ff57e650ef9d None None] PATH_INFO: `/v3` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:27:57.501977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 756/1517] 10.222.0.42 () {54 vars in 925 bytes} [Tue Sep 13 15:27:57 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:27:57.508847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f638036e-7724-4723-88ed-c6f64f3ebb6e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:27:57.509129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f638036e-7724-4723-88ed-c6f64f3ebb6e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:27:57.509396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f638036e-7724-4723-88ed-c6f64f3ebb6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:27:57.577353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-f638036e-7724-4723-88ed-c6f64f3ebb6e None None] MFA Rules not processed for user `4bc6be7644b24ca9bab23128b41a7148`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:27:57.578627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f638036e-7724-4723-88ed-c6f64f3ebb6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:27:57.597107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:27:57.658952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 762/1518] 10.222.0.42 () {58 vars in 1015 bytes} [Tue Sep 13 15:27:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:28:01.352921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df2c5a2-d2f4-4065-a36c-d63f6304e003 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:01.353459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df2c5a2-d2f4-4065-a36c-d63f6304e003 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:01.353459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df2c5a2-d2f4-4065-a36c-d63f6304e003 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:01.355631 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 757/1519] 10.222.0.16 () {54 vars in 961 bytes} [Tue Sep 13 15:28:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:28:01.360949 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c132eee-b75e-4c93-9d0b-9d9df51a51f2 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:01.361365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c132eee-b75e-4c93-9d0b-9d9df51a51f2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:01.361474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c132eee-b75e-4c93-9d0b-9d9df51a51f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:01.387740 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-5c132eee-b75e-4c93-9d0b-9d9df51a51f2 None None] MFA Rules not processed for user `4bc6be7644b24ca9bab23128b41a7148`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:28:01.388940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5c132eee-b75e-4c93-9d0b-9d9df51a51f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:28:01.399145 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 763/1520] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Sep 13 15:28:01 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:28:01.407025 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df2c5a2-d2f4-4065-a36c-d63f6304e003 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:28:01.421547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df2c5a2-d2f4-4065-a36c-d63f6304e003 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:28:01.427270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d1263a-2705-45e2-8537-c7aed02f0510 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:28:01.427866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d1263a-2705-45e2-8537-c7aed02f0510 service placement] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:01.428138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d1263a-2705-45e2-8537-c7aed02f0510 service placement] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:01.428403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d1263a-2705-45e2-8537-c7aed02f0510 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:01.437856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d1263a-2705-45e2-8537-c7aed02f0510 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:28:01.439714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d1263a-2705-45e2-8537-c7aed02f0510 service placement] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:28:01.449765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 758/1521] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:28:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:28:08.485607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d8c06f-ed19-48dd-91c3-36cd04008bbb None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:08.486237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d8c06f-ed19-48dd-91c3-36cd04008bbb None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:08.486237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d8c06f-ed19-48dd-91c3-36cd04008bbb None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:08.487540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 764/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:28:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:28:08.495325 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be8d15-f0a7-4f54-8abc-51a3add1d73e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:08.495807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be8d15-f0a7-4f54-8abc-51a3add1d73e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:08.495919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be8d15-f0a7-4f54-8abc-51a3add1d73e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:08.520750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-44be8d15-f0a7-4f54-8abc-51a3add1d73e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:28:08.521969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-44be8d15-f0a7-4f54-8abc-51a3add1d73e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:28:08.530918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 759/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:28:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:28:08.691125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d8c06f-ed19-48dd-91c3-36cd04008bbb None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:28:08.706410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d8c06f-ed19-48dd-91c3-36cd04008bbb None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:28:08.712336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f76fa72-52bf-45e9-992f-ce691a5cb934 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:28:08.712945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f76fa72-52bf-45e9-992f-ce691a5cb934 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:08.713222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f76fa72-52bf-45e9-992f-ce691a5cb934 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:08.713498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f76fa72-52bf-45e9-992f-ce691a5cb934 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:08.722563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f76fa72-52bf-45e9-992f-ce691a5cb934 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:28:08.724153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f76fa72-52bf-45e9-992f-ce691a5cb934 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:28:08.733786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 765/1524] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:28:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:28:09.814571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5652861c-1ba3-4d1d-8d35-22dcdc979ae3 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:09.815507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5652861c-1ba3-4d1d-8d35-22dcdc979ae3 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:09.815507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5652861c-1ba3-4d1d-8d35-22dcdc979ae3 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:09.817778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 760/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 13 15:28:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:28:09.828060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec715d27-1ec9-4b08-a5ac-cc780e3dc9ee None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:09.828824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec715d27-1ec9-4b08-a5ac-cc780e3dc9ee None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:09.828982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec715d27-1ec9-4b08-a5ac-cc780e3dc9ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:09.851583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-ec715d27-1ec9-4b08-a5ac-cc780e3dc9ee None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:28:09.852809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ec715d27-1ec9-4b08-a5ac-cc780e3dc9ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:28:09.862285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 766/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 13 15:28:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:28:10.018006 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5652861c-1ba3-4d1d-8d35-22dcdc979ae3 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:28:10.034205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5652861c-1ba3-4d1d-8d35-22dcdc979ae3 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:28:10.039956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e51f474-73d9-482e-a268-c621893c3fb8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:28:10.040553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51f474-73d9-482e-a268-c621893c3fb8 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:10.040863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51f474-73d9-482e-a268-c621893c3fb8 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:10.041122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51f474-73d9-482e-a268-c621893c3fb8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:10.049322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e51f474-73d9-482e-a268-c621893c3fb8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:28:10.051037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e51f474-73d9-482e-a268-c621893c3fb8 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:28:10.060757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 761/1527] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:28:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:28:14.169107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bc3ca9-d664-4a12-b803-f958a90f8f64 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:14.170031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bc3ca9-d664-4a12-b803-f958a90f8f64 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:14.170031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bc3ca9-d664-4a12-b803-f958a90f8f64 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:14.172197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 767/1528] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:28:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:28:14.178189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b73805-95c5-42e4-925a-2f9ab79e5e61 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:14.178639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b73805-95c5-42e4-925a-2f9ab79e5e61 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:14.178747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b73805-95c5-42e4-925a-2f9ab79e5e61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:14.198720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-83b73805-95c5-42e4-925a-2f9ab79e5e61 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:28:14.199908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-83b73805-95c5-42e4-925a-2f9ab79e5e61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:28:14.208680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 762/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:28:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:28:14.503259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bc3ca9-d664-4a12-b803-f958a90f8f64 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:28:14.506444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c5bc3ca9-d664-4a12-b803-f958a90f8f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:28:14.597411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bc3ca9-d664-4a12-b803-f958a90f8f64 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:28:14.601960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db302b22-aa0d-41dc-998a-77f67da2537b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:28:14.602628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db302b22-aa0d-41dc-998a-77f67da2537b service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:14.602902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db302b22-aa0d-41dc-998a-77f67da2537b service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:14.603169 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db302b22-aa0d-41dc-998a-77f67da2537b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:14.616292 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db302b22-aa0d-41dc-998a-77f67da2537b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:28:14.618464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db302b22-aa0d-41dc-998a-77f67da2537b service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:28:14.629559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 768/1530] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:28:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:28:16.790146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4a7c65-840a-4cf0-8de7-348be3ab31df None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:16.791081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4a7c65-840a-4cf0-8de7-348be3ab31df None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:16.791081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4a7c65-840a-4cf0-8de7-348be3ab31df None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:16.793246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 763/1531] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:28:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:28:16.798761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8eeecc-ae7c-4f98-b510-6be9cf22b874 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:16.799209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8eeecc-ae7c-4f98-b510-6be9cf22b874 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:16.799312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8eeecc-ae7c-4f98-b510-6be9cf22b874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:16.819841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-cd8eeecc-ae7c-4f98-b510-6be9cf22b874 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:28:16.821044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-cd8eeecc-ae7c-4f98-b510-6be9cf22b874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:28:16.829693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 769/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:28:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:28:17.115748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4a7c65-840a-4cf0-8de7-348be3ab31df None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:28:17.129107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4a7c65-840a-4cf0-8de7-348be3ab31df None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:28:17.132962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f995a2-afc1-4893-96ed-03444b3ae1f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:28:17.133581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f995a2-afc1-4893-96ed-03444b3ae1f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:17.133862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f995a2-afc1-4893-96ed-03444b3ae1f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:17.134158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f995a2-afc1-4893-96ed-03444b3ae1f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:17.143225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f995a2-afc1-4893-96ed-03444b3ae1f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:28:17.144868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f995a2-afc1-4893-96ed-03444b3ae1f5 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:28:17.154329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 764/1533] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:28:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:28:21.385071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9c24d-0ad4-4139-a56f-a479482475e7 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:21.385662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9c24d-0ad4-4139-a56f-a479482475e7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:21.385662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9c24d-0ad4-4139-a56f-a479482475e7 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:21.387001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 770/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:28:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:28:21.392559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ddbf6-82f6-440a-a263-bcebf0472245 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:21.393007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ddbf6-82f6-440a-a263-bcebf0472245 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:21.393110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ddbf6-82f6-440a-a263-bcebf0472245 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:21.414149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7b2ddbf6-82f6-440a-a263-bcebf0472245 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:28:21.415348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7b2ddbf6-82f6-440a-a263-bcebf0472245 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:28:21.423758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 765/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:28:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:28:21.719845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a9c24d-0ad4-4139-a56f-a479482475e7 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:28:21.722004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-63a9c24d-0ad4-4139-a56f-a479482475e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:28:21.791824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:28:21.808443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a9c24d-0ad4-4139-a56f-a479482475e7 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:28:21.813395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e50441f-451d-45f8-ac00-f6fa819a8a93 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:28:21.814010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e50441f-451d-45f8-ac00-f6fa819a8a93 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:21.814314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e50441f-451d-45f8-ac00-f6fa819a8a93 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:21.814600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e50441f-451d-45f8-ac00-f6fa819a8a93 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:21.825066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e50441f-451d-45f8-ac00-f6fa819a8a93 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:28:21.826522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e50441f-451d-45f8-ac00-f6fa819a8a93 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:28:21.836387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 771/1536] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:28:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:28:23.980154 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbf3bed-c105-4a4e-ae3b-6dd8d441af5a None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:23.980726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbf3bed-c105-4a4e-ae3b-6dd8d441af5a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:23.980726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbf3bed-c105-4a4e-ae3b-6dd8d441af5a None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:23.982952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 766/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 13 15:28:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:28:23.990498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496775f6-8690-4ca4-8391-6bbbf56112ba None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:23.991007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496775f6-8690-4ca4-8391-6bbbf56112ba None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:23.991007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496775f6-8690-4ca4-8391-6bbbf56112ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:24.012149 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-496775f6-8690-4ca4-8391-6bbbf56112ba None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:28:24.013370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-496775f6-8690-4ca4-8391-6bbbf56112ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:28:24.023207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 772/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 13 15:28:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:28:24.320070 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbf3bed-c105-4a4e-ae3b-6dd8d441af5a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:28:24.322188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7dbf3bed-c105-4a4e-ae3b-6dd8d441af5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:28:24.338642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbf3bed-c105-4a4e-ae3b-6dd8d441af5a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:28:24.342925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7adddd-3f46-45ec-ae72-61666fece579 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:28:24.343521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7adddd-3f46-45ec-ae72-61666fece579 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:28:24.343799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7adddd-3f46-45ec-ae72-61666fece579 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:28:24.344064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7adddd-3f46-45ec-ae72-61666fece579 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:28:24.353886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7adddd-3f46-45ec-ae72-61666fece579 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:28:24.355368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7adddd-3f46-45ec-ae72-61666fece579 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:28:24.365041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 767/1539] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:28:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:28:34.006807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=11 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:28:34.360644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:48.982018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf3e50-1927-4d4b-8380-23fd88a1206d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:48.983562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf3e50-1927-4d4b-8380-23fd88a1206d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:48.983562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cf3e50-1927-4d4b-8380-23fd88a1206d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.010546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e6cf3e50-1927-4d4b-8380-23fd88a1206d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.011791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e6cf3e50-1927-4d4b-8380-23fd88a1206d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.027201 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 773/1540] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.041241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ae0a5c-396d-4dc7-ab22-c18310cd34c9 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.041241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ae0a5c-396d-4dc7-ab22-c18310cd34c9 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.041973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ae0a5c-396d-4dc7-ab22-c18310cd34c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.042264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e29186-0494-4013-874f-d23893bd0064 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.042507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e29186-0494-4013-874f-d23893bd0064 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.042780 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e29186-0494-4013-874f-d23893bd0064 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.072220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-78ae0a5c-396d-4dc7-ab22-c18310cd34c9 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.073544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-78ae0a5c-396d-4dc7-ab22-c18310cd34c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.074734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-43e29186-0494-4013-874f-d23893bd0064 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.075925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-43e29186-0494-4013-874f-d23893bd0064 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.088112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 768/1541] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:29:49.089462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb2c7e4-72e8-4ad9-8411-8633128edb1f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.089751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb2c7e4-72e8-4ad9-8411-8633128edb1f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.090020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb2c7e4-72e8-4ad9-8411-8633128edb1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.100372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 774/1542] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:29:49.101927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d9e3c-5dfc-42b7-9284-128084470b98 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.102352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d9e3c-5dfc-42b7-9284-128084470b98 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.102630 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d9e3c-5dfc-42b7-9284-128084470b98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.121030 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8cb2c7e4-72e8-4ad9-8411-8633128edb1f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.122598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8cb2c7e4-72e8-4ad9-8411-8633128edb1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.135330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-033d9e3c-5dfc-42b7-9284-128084470b98 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.136570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 769/1543] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.137212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-033d9e3c-5dfc-42b7-9284-128084470b98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.137875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638fb2e-d700-4f5f-b552-f14fe23b8f6f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.138445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638fb2e-d700-4f5f-b552-f14fe23b8f6f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.138683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638fb2e-d700-4f5f-b552-f14fe23b8f6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.159866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 775/1544] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.160878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033d9e3c-5dfc-42b7-9284-128084470b98 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.163160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2638fb2e-d700-4f5f-b552-f14fe23b8f6f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.164778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2638fb2e-d700-4f5f-b552-f14fe23b8f6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.174771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033d9e3c-5dfc-42b7-9284-128084470b98 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.181514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 770/1545] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.182991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407a3fca-9b59-4328-88aa-673615125e9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.185891 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407a3fca-9b59-4328-88aa-673615125e9e admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.186458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407a3fca-9b59-4328-88aa-673615125e9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.187002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407a3fca-9b59-4328-88aa-673615125e9e admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.187463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b6b3d-5341-4633-a4c2-5c855454a0a5 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.188333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407a3fca-9b59-4328-88aa-673615125e9e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:29:49.188829 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b6b3d-5341-4633-a4c2-5c855454a0a5 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.189303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b6b3d-5341-4633-a4c2-5c855454a0a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.189814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407a3fca-9b59-4328-88aa-673615125e9e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.194327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407a3fca-9b59-4328-88aa-673615125e9e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.222163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 776/1546] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:29:49.224253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a560672-4223-4b37-8fb8-8c3ef64e31ab None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.224537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a560672-4223-4b37-8fb8-8c3ef64e31ab None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.224827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a560672-4223-4b37-8fb8-8c3ef64e31ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.232256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-ae4b6b3d-5341-4633-a4c2-5c855454a0a5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.233425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ae4b6b3d-5341-4633-a4c2-5c855454a0a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.242769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 771/1547] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.244238 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad8b27-047e-4f81-9a86-d579ae034f5a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.244516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad8b27-047e-4f81-9a86-d579ae034f5a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.244781 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad8b27-047e-4f81-9a86-d579ae034f5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.255032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0a560672-4223-4b37-8fb8-8c3ef64e31ab None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.256244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0a560672-4223-4b37-8fb8-8c3ef64e31ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.265760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-83ad8b27-047e-4f81-9a86-d579ae034f5a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.266458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 777/1548] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.267059 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-83ad8b27-047e-4f81-9a86-d579ae034f5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.267275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a560672-4223-4b37-8fb8-8c3ef64e31ab None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.279610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 772/1549] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.280105 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a560672-4223-4b37-8fb8-8c3ef64e31ab None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.281176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122602a0-2cb3-40ad-b529-bd16aaffb40d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.281480 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122602a0-2cb3-40ad-b529-bd16aaffb40d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.281749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122602a0-2cb3-40ad-b529-bd16aaffb40d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.285407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d89a0c-d346-4c7f-83f0-e83d5fee4b8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.286015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d89a0c-d346-4c7f-83f0-e83d5fee4b8a admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.286321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d89a0c-d346-4c7f-83f0-e83d5fee4b8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.286590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d89a0c-d346-4c7f-83f0-e83d5fee4b8a admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.287236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d89a0c-d346-4c7f-83f0-e83d5fee4b8a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:29:49.287537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d89a0c-d346-4c7f-83f0-e83d5fee4b8a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.289129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d89a0c-d346-4c7f-83f0-e83d5fee4b8a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.301916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-122602a0-2cb3-40ad-b529-bd16aaffb40d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.302485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 778/1550] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:29:49.303151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-122602a0-2cb3-40ad-b529-bd16aaffb40d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.303311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d89a0c-d346-4c7f-83f0-e83d5fee4b8a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.312241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 773/1551] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.313843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d89a0c-d346-4c7f-83f0-e83d5fee4b8a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.315450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dff7e1-919a-4a1b-a7f1-ef02ebdaf035 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.315736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dff7e1-919a-4a1b-a7f1-ef02ebdaf035 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.316021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dff7e1-919a-4a1b-a7f1-ef02ebdaf035 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.318612 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72720094-13d2-418f-9efd-31b6b8c3956e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.319476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72720094-13d2-418f-9efd-31b6b8c3956e admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.319757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72720094-13d2-418f-9efd-31b6b8c3956e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.320022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72720094-13d2-418f-9efd-31b6b8c3956e admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.320651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72720094-13d2-418f-9efd-31b6b8c3956e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.321653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: WARNING py.warnings [None req-72720094-13d2-418f-9efd-31b6b8c3956e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:29:49.321653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: warnings.warn(msg) Sep 13 15:29:49.321653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: Sep 13 15:29:49.322851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72720094-13d2-418f-9efd-31b6b8c3956e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.335605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-04dff7e1-919a-4a1b-a7f1-ef02ebdaf035 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.336822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-04dff7e1-919a-4a1b-a7f1-ef02ebdaf035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.344948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 774/1552] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.346209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e0b84-07fb-4fbc-adb0-1daecbf36c95 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.346486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e0b84-07fb-4fbc-adb0-1daecbf36c95 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.346741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e0b84-07fb-4fbc-adb0-1daecbf36c95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.363854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 779/1553] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/projects => generated 375 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:49.367618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72720094-13d2-418f-9efd-31b6b8c3956e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.372744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8a1e0b84-07fb-4fbc-adb0-1daecbf36c95 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.374384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8a1e0b84-07fb-4fbc-adb0-1daecbf36c95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.427708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72720094-13d2-418f-9efd-31b6b8c3956e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.432898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a411cd-4ecc-45e5-ab67-9f71d8fed61e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.433516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a411cd-4ecc-45e5-ab67-9f71d8fed61e admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.433795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a411cd-4ecc-45e5-ab67-9f71d8fed61e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.434092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a411cd-4ecc-45e5-ab67-9f71d8fed61e admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.435313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a411cd-4ecc-45e5-ab67-9f71d8fed61e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:29:49.435616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a411cd-4ecc-45e5-ab67-9f71d8fed61e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.437235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a411cd-4ecc-45e5-ab67-9f71d8fed61e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.438326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 775/1554] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.439492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb3ee81-3d9f-4280-8bd4-635aab13bd83 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.439777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb3ee81-3d9f-4280-8bd4-635aab13bd83 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.440059 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb3ee81-3d9f-4280-8bd4-635aab13bd83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.449792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 780/1555] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:29:49.450994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a411cd-4ecc-45e5-ab67-9f71d8fed61e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.463081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-cdb3ee81-3d9f-4280-8bd4-635aab13bd83 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.464292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-cdb3ee81-3d9f-4280-8bd4-635aab13bd83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.470719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a411cd-4ecc-45e5-ab67-9f71d8fed61e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.472772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 776/1556] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.473657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb3ee81-3d9f-4280-8bd4-635aab13bd83 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.477018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4278ab3a-7f2b-49da-a2bf-927321a90b5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.477700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4278ab3a-7f2b-49da-a2bf-927321a90b5d admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.478011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4278ab3a-7f2b-49da-a2bf-927321a90b5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.478360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4278ab3a-7f2b-49da-a2bf-927321a90b5d admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.479077 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4278ab3a-7f2b-49da-a2bf-927321a90b5d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:29:49.479433 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4278ab3a-7f2b-49da-a2bf-927321a90b5d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.481158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4278ab3a-7f2b-49da-a2bf-927321a90b5d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.484009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb3ee81-3d9f-4280-8bd4-635aab13bd83 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.488452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef8c63f-3b53-4b03-a869-46ea5b524270 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.489050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef8c63f-3b53-4b03-a869-46ea5b524270 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.489311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef8c63f-3b53-4b03-a869-46ea5b524270 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.489577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef8c63f-3b53-4b03-a869-46ea5b524270 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.490230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef8c63f-3b53-4b03-a869-46ea5b524270 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.492125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef8c63f-3b53-4b03-a869-46ea5b524270 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.494384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 781/1557] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:29:49.495846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e74293-25b6-4372-993a-bf218c9fa3d6 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.496115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e74293-25b6-4372-993a-bf218c9fa3d6 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.496376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e74293-25b6-4372-993a-bf218c9fa3d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.517059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-75e74293-25b6-4372-993a-bf218c9fa3d6 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.518611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-75e74293-25b6-4372-993a-bf218c9fa3d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.527257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 782/1558] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.527671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 777/1559] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/projects => generated 379 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:49.528510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da8d3eb-6789-4c4a-94e8-801a45b9b000 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.528626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef8c63f-3b53-4b03-a869-46ea5b524270 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.528784 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da8d3eb-6789-4c4a-94e8-801a45b9b000 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.529053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da8d3eb-6789-4c4a-94e8-801a45b9b000 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.548515 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-9da8d3eb-6789-4c4a-94e8-801a45b9b000 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.549734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9da8d3eb-6789-4c4a-94e8-801a45b9b000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.596965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:49.599990 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef8c63f-3b53-4b03-a869-46ea5b524270 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.605609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0c3e39-1e40-4a9d-8d7a-54547bc5b42e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.606328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0c3e39-1e40-4a9d-8d7a-54547bc5b42e admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.606613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0c3e39-1e40-4a9d-8d7a-54547bc5b42e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.606888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0c3e39-1e40-4a9d-8d7a-54547bc5b42e admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.607528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0c3e39-1e40-4a9d-8d7a-54547bc5b42e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.608613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: WARNING py.warnings [None req-4a0c3e39-1e40-4a9d-8d7a-54547bc5b42e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:29:49.608613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: warnings.warn(msg) Sep 13 15:29:49.608613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: Sep 13 15:29:49.609828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0c3e39-1e40-4a9d-8d7a-54547bc5b42e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.617633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 783/1560] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.617633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da8d3eb-6789-4c4a-94e8-801a45b9b000 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.624813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da8d3eb-6789-4c4a-94e8-801a45b9b000 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.629723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ada82a-4a55-4a4a-bc8c-0a45a00a06aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.630373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ada82a-4a55-4a4a-bc8c-0a45a00a06aa admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.630651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ada82a-4a55-4a4a-bc8c-0a45a00a06aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.630947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ada82a-4a55-4a4a-bc8c-0a45a00a06aa admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.631595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ada82a-4a55-4a4a-bc8c-0a45a00a06aa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:29:49.631897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ada82a-4a55-4a4a-bc8c-0a45a00a06aa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.633509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ada82a-4a55-4a4a-bc8c-0a45a00a06aa admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.633904 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 778/1561] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/users => generated 288 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:49.634823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0c3e39-1e40-4a9d-8d7a-54547bc5b42e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.649334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0c3e39-1e40-4a9d-8d7a-54547bc5b42e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.650412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 784/1562] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:29:49.651713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dd8f0d-b3ef-47f7-b09b-a492350cdcea None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.652101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dd8f0d-b3ef-47f7-b09b-a492350cdcea None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.652380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dd8f0d-b3ef-47f7-b09b-a492350cdcea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.654309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de33f82-c5d6-4ef6-bdda-c0bb600f901a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.654914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de33f82-c5d6-4ef6-bdda-c0bb600f901a admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.655188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de33f82-c5d6-4ef6-bdda-c0bb600f901a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.655457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de33f82-c5d6-4ef6-bdda-c0bb600f901a admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.656079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de33f82-c5d6-4ef6-bdda-c0bb600f901a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.657972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de33f82-c5d6-4ef6-bdda-c0bb600f901a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.673062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a9dd8f0d-b3ef-47f7-b09b-a492350cdcea None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:49.673248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:49.674355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a9dd8f0d-b3ef-47f7-b09b-a492350cdcea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:49.683915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 785/1563] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:49.684873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dd8f0d-b3ef-47f7-b09b-a492350cdcea None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.692844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 779/1564] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/projects => generated 377 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:49.693979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de33f82-c5d6-4ef6-bdda-c0bb600f901a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.697255 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dd8f0d-b3ef-47f7-b09b-a492350cdcea None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.753839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3412d177-8610-4b84-80eb-bfd294a784e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.754395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3412d177-8610-4b84-80eb-bfd294a784e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.754671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3412d177-8610-4b84-80eb-bfd294a784e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.754935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3412d177-8610-4b84-80eb-bfd294a784e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.755567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3412d177-8610-4b84-80eb-bfd294a784e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.757380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de33f82-c5d6-4ef6-bdda-c0bb600f901a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.757632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3412d177-8610-4b84-80eb-bfd294a784e9 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.761883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c813f283-c54c-487e-842b-0be16df8ab61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.762874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c813f283-c54c-487e-842b-0be16df8ab61 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.762874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c813f283-c54c-487e-842b-0be16df8ab61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.763196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c813f283-c54c-487e-842b-0be16df8ab61 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.763877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c813f283-c54c-487e-842b-0be16df8ab61 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:29:49.764305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c813f283-c54c-487e-842b-0be16df8ab61 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.765337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: WARNING py.warnings [None req-c813f283-c54c-487e-842b-0be16df8ab61 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:29:49.765337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: warnings.warn(msg) Sep 13 15:29:49.765337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: Sep 13 15:29:49.766301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c813f283-c54c-487e-842b-0be16df8ab61 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.778993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 780/1565] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:29:49.779896 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c813f283-c54c-487e-842b-0be16df8ab61 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.794738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 786/1566] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/projects => generated 375 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:49.794738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3412d177-8610-4b84-80eb-bfd294a784e9 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.795044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c813f283-c54c-487e-842b-0be16df8ab61 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.854763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2059250-e2b7-48fe-95f1-aaee3335df51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.855380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2059250-e2b7-48fe-95f1-aaee3335df51 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.855687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2059250-e2b7-48fe-95f1-aaee3335df51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.855984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2059250-e2b7-48fe-95f1-aaee3335df51 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.857145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2059250-e2b7-48fe-95f1-aaee3335df51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.859157 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2059250-e2b7-48fe-95f1-aaee3335df51 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.862457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3412d177-8610-4b84-80eb-bfd294a784e9 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.867363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67c51b-cc09-416a-ba85-09e4e603f2c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.868083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67c51b-cc09-416a-ba85-09e4e603f2c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.868424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67c51b-cc09-416a-ba85-09e4e603f2c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.868846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67c51b-cc09-416a-ba85-09e4e603f2c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.869791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf67c51b-cc09-416a-ba85-09e4e603f2c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:29:49.870354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf67c51b-cc09-416a-ba85-09e4e603f2c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.872549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf67c51b-cc09-416a-ba85-09e4e603f2c4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.880930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 781/1567] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/users => generated 290 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:49.881791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2059250-e2b7-48fe-95f1-aaee3335df51 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.886087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 787/1568] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:29:49.886908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67c51b-cc09-416a-ba85-09e4e603f2c4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.895118 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2059250-e2b7-48fe-95f1-aaee3335df51 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.896809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67c51b-cc09-416a-ba85-09e4e603f2c4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.899920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ccbbbc-73fd-4330-a087-b2521dff27ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.900520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccbbbc-73fd-4330-a087-b2521dff27ff admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.900791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccbbbc-73fd-4330-a087-b2521dff27ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.901053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ccbbbc-73fd-4330-a087-b2521dff27ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.901390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e67c9b-b8b4-4917-a853-5f8c7171df28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.901686 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ccbbbc-73fd-4330-a087-b2521dff27ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.901978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e67c9b-b8b4-4917-a853-5f8c7171df28 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.902289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e67c9b-b8b4-4917-a853-5f8c7171df28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.902557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e67c9b-b8b4-4917-a853-5f8c7171df28 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.903923 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e67c9b-b8b4-4917-a853-5f8c7171df28 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.904824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ccbbbc-73fd-4330-a087-b2521dff27ff admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.905805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e67c9b-b8b4-4917-a853-5f8c7171df28 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.918573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 782/1569] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:49 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:49.918573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ccbbbc-73fd-4330-a087-b2521dff27ff admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.927222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ccbbbc-73fd-4330-a087-b2521dff27ff admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.932673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8efdb50-bccb-42cb-a778-c54870b7b031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:49.933259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8efdb50-bccb-42cb-a778-c54870b7b031 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:49.933533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8efdb50-bccb-42cb-a778-c54870b7b031 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:49.933796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8efdb50-bccb-42cb-a778-c54870b7b031 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:49.934547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8efdb50-bccb-42cb-a778-c54870b7b031 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:29:49.934890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8efdb50-bccb-42cb-a778-c54870b7b031 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:49.936604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 788/1570] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:49.936720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8efdb50-bccb-42cb-a778-c54870b7b031 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:49.937454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e67c9b-b8b4-4917-a853-5f8c7171df28 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.956554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 783/1571] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:29:49.956554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8efdb50-bccb-42cb-a778-c54870b7b031 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:49.960210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8efdb50-bccb-42cb-a778-c54870b7b031 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:49.974080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:50.010423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e67c9b-b8b4-4917-a853-5f8c7171df28 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.015418 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:50.015598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4af953-21b1-4884-a105-74b8abc16ebf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.016173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4af953-21b1-4884-a105-74b8abc16ebf admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.016475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4af953-21b1-4884-a105-74b8abc16ebf admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.016864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4af953-21b1-4884-a105-74b8abc16ebf admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.017380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4af953-21b1-4884-a105-74b8abc16ebf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.019601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4af953-21b1-4884-a105-74b8abc16ebf admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.025036 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592e8eb8-263c-41a2-badc-e04424019a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.025652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e8eb8-263c-41a2-badc-e04424019a3d admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.025930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e8eb8-263c-41a2-badc-e04424019a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.026240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e8eb8-263c-41a2-badc-e04424019a3d admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.026875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592e8eb8-263c-41a2-badc-e04424019a3d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.029004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592e8eb8-263c-41a2-badc-e04424019a3d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.045051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 789/1572] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/users => generated 289 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:50.045870 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4af953-21b1-4884-a105-74b8abc16ebf admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.058324 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4af953-21b1-4884-a105-74b8abc16ebf admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.068080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 784/1573] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:49 2022] POST /identity/v3/projects => generated 385 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:50.068666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99475acf-e235-4d91-854f-a9086e2e0b61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.068666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99475acf-e235-4d91-854f-a9086e2e0b61 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.068875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592e8eb8-263c-41a2-badc-e04424019a3d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.068975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99475acf-e235-4d91-854f-a9086e2e0b61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.068975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99475acf-e235-4d91-854f-a9086e2e0b61 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.068975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99475acf-e235-4d91-854f-a9086e2e0b61 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.071522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99475acf-e235-4d91-854f-a9086e2e0b61 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.103768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 790/1574] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:50 2022] POST /identity/v3/users => generated 288 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:50.104691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99475acf-e235-4d91-854f-a9086e2e0b61 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.115588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99475acf-e235-4d91-854f-a9086e2e0b61 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.144588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592e8eb8-263c-41a2-badc-e04424019a3d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.156258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7817230-7ed6-409a-b858-0ef6f9aef2a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.156897 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7817230-7ed6-409a-b858-0ef6f9aef2a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.157170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7817230-7ed6-409a-b858-0ef6f9aef2a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.157430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7817230-7ed6-409a-b858-0ef6f9aef2a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.158117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7817230-7ed6-409a-b858-0ef6f9aef2a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.160195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7817230-7ed6-409a-b858-0ef6f9aef2a5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.171323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 785/1575] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.171967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7817230-7ed6-409a-b858-0ef6f9aef2a5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.181747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7817230-7ed6-409a-b858-0ef6f9aef2a5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.183924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e645bb0f-548b-45cc-8faf-988293335be2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.185135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e645bb0f-548b-45cc-8faf-988293335be2 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.185410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e645bb0f-548b-45cc-8faf-988293335be2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.185675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e645bb0f-548b-45cc-8faf-988293335be2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.186396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e645bb0f-548b-45cc-8faf-988293335be2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.186652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69824a8-cf2b-4944-a82d-c891af383770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.187251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69824a8-cf2b-4944-a82d-c891af383770 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.187511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69824a8-cf2b-4944-a82d-c891af383770 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.187781 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69824a8-cf2b-4944-a82d-c891af383770 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.188283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e645bb0f-548b-45cc-8faf-988293335be2 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.188397 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69824a8-cf2b-4944-a82d-c891af383770 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.190000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69824a8-cf2b-4944-a82d-c891af383770 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.201179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 786/1576] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.201977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69824a8-cf2b-4944-a82d-c891af383770 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.212587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69824a8-cf2b-4944-a82d-c891af383770 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.217473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea5f19c-f8d5-46e9-b5b0-0fb219f4e229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.218021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 791/1577] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:50 2022] POST /identity/v3/projects => generated 389 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:50.218159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea5f19c-f8d5-46e9-b5b0-0fb219f4e229 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.218411 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea5f19c-f8d5-46e9-b5b0-0fb219f4e229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.218644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea5f19c-f8d5-46e9-b5b0-0fb219f4e229 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.218739 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e645bb0f-548b-45cc-8faf-988293335be2 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.219275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea5f19c-f8d5-46e9-b5b0-0fb219f4e229 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.221478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea5f19c-f8d5-46e9-b5b0-0fb219f4e229 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.246134 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 787/1578] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:50 2022] POST /identity/v3/users => generated 287 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:50.246884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea5f19c-f8d5-46e9-b5b0-0fb219f4e229 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.256748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea5f19c-f8d5-46e9-b5b0-0fb219f4e229 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.285269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e645bb0f-548b-45cc-8faf-988293335be2 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.289942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14eaab7-38d4-4c24-8a03-1ab084c03234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.290583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14eaab7-38d4-4c24-8a03-1ab084c03234 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.290853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14eaab7-38d4-4c24-8a03-1ab084c03234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.291812 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14eaab7-38d4-4c24-8a03-1ab084c03234 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.291812 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14eaab7-38d4-4c24-8a03-1ab084c03234 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.293654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14eaab7-38d4-4c24-8a03-1ab084c03234 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.314636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cd1447-0943-4801-9add-2715faf9076a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.315255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cd1447-0943-4801-9add-2715faf9076a admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.315535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cd1447-0943-4801-9add-2715faf9076a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.315813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cd1447-0943-4801-9add-2715faf9076a admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.316461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cd1447-0943-4801-9add-2715faf9076a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.318082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cd1447-0943-4801-9add-2715faf9076a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.325381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 792/1579] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:50 2022] POST /identity/v3/projects => generated 393 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:50.325381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14eaab7-38d4-4c24-8a03-1ab084c03234 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.330928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 788/1580] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.330928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cd1447-0943-4801-9add-2715faf9076a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.336398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:50.339193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cd1447-0943-4801-9add-2715faf9076a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.380249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:50.387958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14eaab7-38d4-4c24-8a03-1ab084c03234 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.392655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa051fd-92b5-44c6-82e9-9f249aa0ba7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.393514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa051fd-92b5-44c6-82e9-9f249aa0ba7c admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.393742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa051fd-92b5-44c6-82e9-9f249aa0ba7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.394095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa051fd-92b5-44c6-82e9-9f249aa0ba7c admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.394925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa051fd-92b5-44c6-82e9-9f249aa0ba7c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.397223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa051fd-92b5-44c6-82e9-9f249aa0ba7c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.397496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc80598-a5d9-4950-9de4-7da4ad02369b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.398093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc80598-a5d9-4950-9de4-7da4ad02369b admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.398391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc80598-a5d9-4950-9de4-7da4ad02369b admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.398775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc80598-a5d9-4950-9de4-7da4ad02369b admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.399426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc80598-a5d9-4950-9de4-7da4ad02369b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.401021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc80598-a5d9-4950-9de4-7da4ad02369b admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.411261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 789/1581] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.411824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc80598-a5d9-4950-9de4-7da4ad02369b admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.421933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc80598-a5d9-4950-9de4-7da4ad02369b admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.428558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 793/1582] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:50 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:50.428558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa051fd-92b5-44c6-82e9-9f249aa0ba7c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.429463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09d8768-9749-4305-8f8f-7efd2e52e8ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.430068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09d8768-9749-4305-8f8f-7efd2e52e8ba admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.430397 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09d8768-9749-4305-8f8f-7efd2e52e8ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.430695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09d8768-9749-4305-8f8f-7efd2e52e8ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.431360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09d8768-9749-4305-8f8f-7efd2e52e8ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.432967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09d8768-9749-4305-8f8f-7efd2e52e8ba admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.438677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa051fd-92b5-44c6-82e9-9f249aa0ba7c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.443170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 790/1583] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.443884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557fed86-91ab-4add-b614-b1950cca5a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.444570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557fed86-91ab-4add-b614-b1950cca5a90 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.444871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557fed86-91ab-4add-b614-b1950cca5a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.445128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557fed86-91ab-4add-b614-b1950cca5a90 admin admin] PATH_INFO: `/v3/projects/597b9e023e324874ac5f24ee571d1f66/users/c02aec8271af4bf7aea728ff92fa555b/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.447033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09d8768-9749-4305-8f8f-7efd2e52e8ba admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.456751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09d8768-9749-4305-8f8f-7efd2e52e8ba admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.461609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557fed86-91ab-4add-b614-b1950cca5a90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=597b9e023e324874ac5f24ee571d1f66, user_id=c02aec8271af4bf7aea728ff92fa555b, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.462435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae77ef89-6f3f-48eb-8bbf-efaa13a6cff5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.462931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: WARNING py.warnings [None req-557fed86-91ab-4add-b614-b1950cca5a90 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:29:50.462931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: warnings.warn(msg) Sep 13 15:29:50.462931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: Sep 13 15:29:50.463220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae77ef89-6f3f-48eb-8bbf-efaa13a6cff5 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.463323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae77ef89-6f3f-48eb-8bbf-efaa13a6cff5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.463563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae77ef89-6f3f-48eb-8bbf-efaa13a6cff5 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.464251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae77ef89-6f3f-48eb-8bbf-efaa13a6cff5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.464859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557fed86-91ab-4add-b614-b1950cca5a90 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.466527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae77ef89-6f3f-48eb-8bbf-efaa13a6cff5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.479124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 794/1584] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:50 2022] PUT /identity/v3/projects/597b9e023e324874ac5f24ee571d1f66/users/c02aec8271af4bf7aea728ff92fa555b/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:50.479905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557fed86-91ab-4add-b614-b1950cca5a90 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.495335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 791/1585] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:50 2022] POST /identity/v3/users => generated 295 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:50.496162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae77ef89-6f3f-48eb-8bbf-efaa13a6cff5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.506919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae77ef89-6f3f-48eb-8bbf-efaa13a6cff5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.542263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557fed86-91ab-4add-b614-b1950cca5a90 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.546909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69844222-8c47-4af7-931c-6a0fdf2ece5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.547523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69844222-8c47-4af7-931c-6a0fdf2ece5d admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.547800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69844222-8c47-4af7-931c-6a0fdf2ece5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.548069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69844222-8c47-4af7-931c-6a0fdf2ece5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.548788 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69844222-8c47-4af7-931c-6a0fdf2ece5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.550572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69844222-8c47-4af7-931c-6a0fdf2ece5d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.563495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 795/1586] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.563495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69844222-8c47-4af7-931c-6a0fdf2ece5d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.566765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4247914-6e3e-47d4-8c33-e26544fd9ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.567377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4247914-6e3e-47d4-8c33-e26544fd9ef2 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.567686 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4247914-6e3e-47d4-8c33-e26544fd9ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.567931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4247914-6e3e-47d4-8c33-e26544fd9ef2 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.568574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4247914-6e3e-47d4-8c33-e26544fd9ef2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.570604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4247914-6e3e-47d4-8c33-e26544fd9ef2 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.570750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69844222-8c47-4af7-931c-6a0fdf2ece5d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.575108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79171c9-d648-4e00-b88b-47c27d14dfa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.575794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79171c9-d648-4e00-b88b-47c27d14dfa5 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.576097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79171c9-d648-4e00-b88b-47c27d14dfa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.576369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79171c9-d648-4e00-b88b-47c27d14dfa5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.576995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79171c9-d648-4e00-b88b-47c27d14dfa5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.578605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79171c9-d648-4e00-b88b-47c27d14dfa5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.588960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 796/1587] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.593012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79171c9-d648-4e00-b88b-47c27d14dfa5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.593538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 792/1588] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:50 2022] POST /identity/v3/users => generated 297 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:50.594204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4247914-6e3e-47d4-8c33-e26544fd9ef2 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.603532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79171c9-d648-4e00-b88b-47c27d14dfa5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.604022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4247914-6e3e-47d4-8c33-e26544fd9ef2 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.608141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29b258c-5f24-424b-9b1d-e0f65570038c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.608731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29b258c-5f24-424b-9b1d-e0f65570038c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.608846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce37203a-0e2e-4c3b-84d1-568077f52845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.609004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29b258c-5f24-424b-9b1d-e0f65570038c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.609270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29b258c-5f24-424b-9b1d-e0f65570038c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.609379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37203a-0e2e-4c3b-84d1-568077f52845 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.609616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37203a-0e2e-4c3b-84d1-568077f52845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.609887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37203a-0e2e-4c3b-84d1-568077f52845 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.610008 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29b258c-5f24-424b-9b1d-e0f65570038c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.610563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce37203a-0e2e-4c3b-84d1-568077f52845 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.611541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29b258c-5f24-424b-9b1d-e0f65570038c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.612144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce37203a-0e2e-4c3b-84d1-568077f52845 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.622134 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 793/1589] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.622776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce37203a-0e2e-4c3b-84d1-568077f52845 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.624883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 797/1590] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.626470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23daf9ec-b66b-4507-826c-a02988d66f18 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.626757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23daf9ec-b66b-4507-826c-a02988d66f18 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.627028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23daf9ec-b66b-4507-826c-a02988d66f18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.635252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce37203a-0e2e-4c3b-84d1-568077f52845 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.641947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92db1bde-322c-4071-b6a8-adde6d5a5b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.642671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92db1bde-322c-4071-b6a8-adde6d5a5b50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.642954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92db1bde-322c-4071-b6a8-adde6d5a5b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.643275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92db1bde-322c-4071-b6a8-adde6d5a5b50 admin admin] PATH_INFO: `/v3/projects/3db54de0d87e4f9ea30a70cfddfd595e/users/7c4f9efdfd2d498d8f76360bb17fd021/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.645860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-23daf9ec-b66b-4507-826c-a02988d66f18 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:50.647159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-23daf9ec-b66b-4507-826c-a02988d66f18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:50.655529 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 798/1591] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:50.656387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23daf9ec-b66b-4507-826c-a02988d66f18 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.660565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92db1bde-322c-4071-b6a8-adde6d5a5b50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3db54de0d87e4f9ea30a70cfddfd595e, user_id=7c4f9efdfd2d498d8f76360bb17fd021, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.663606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92db1bde-322c-4071-b6a8-adde6d5a5b50 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.666785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23daf9ec-b66b-4507-826c-a02988d66f18 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.672783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3ed2d1-051e-4bc8-8686-2c84d94b079c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.673497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3ed2d1-051e-4bc8-8686-2c84d94b079c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.673867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3ed2d1-051e-4bc8-8686-2c84d94b079c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.674224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3ed2d1-051e-4bc8-8686-2c84d94b079c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.674928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3ed2d1-051e-4bc8-8686-2c84d94b079c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.676524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3ed2d1-051e-4bc8-8686-2c84d94b079c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.677854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 794/1592] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:50 2022] PUT /identity/v3/projects/3db54de0d87e4f9ea30a70cfddfd595e/users/7c4f9efdfd2d498d8f76360bb17fd021/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:50.678714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92db1bde-322c-4071-b6a8-adde6d5a5b50 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.695796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 799/1593] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.697047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3ed2d1-051e-4bc8-8686-2c84d94b079c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.707434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3ed2d1-051e-4bc8-8686-2c84d94b079c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.737272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:50.741581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92db1bde-322c-4071-b6a8-adde6d5a5b50 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.746635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaed9138-dfae-48a8-9c7a-beb48768c5f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.747250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaed9138-dfae-48a8-9c7a-beb48768c5f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.747536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaed9138-dfae-48a8-9c7a-beb48768c5f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.747806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaed9138-dfae-48a8-9c7a-beb48768c5f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.748449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaed9138-dfae-48a8-9c7a-beb48768c5f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.750319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaed9138-dfae-48a8-9c7a-beb48768c5f5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.755897 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:50.766293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 795/1594] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.767320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaed9138-dfae-48a8-9c7a-beb48768c5f5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.768589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79b72cd-0e76-4836-b1a4-9c09aea1252f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.769753 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79b72cd-0e76-4836-b1a4-9c09aea1252f admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.770286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79b72cd-0e76-4836-b1a4-9c09aea1252f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.770733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79b72cd-0e76-4836-b1a4-9c09aea1252f admin admin] PATH_INFO: `/v3/projects/836a885e5c97498db1721adb1a22a05f/users/c8dcc087badb43b7a18d294a6e316fe0/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.778660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaed9138-dfae-48a8-9c7a-beb48768c5f5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.783451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88867ff9-6774-405d-b483-04d5790dda5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.784067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88867ff9-6774-405d-b483-04d5790dda5f admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.784349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88867ff9-6774-405d-b483-04d5790dda5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.784619 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88867ff9-6774-405d-b483-04d5790dda5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.785263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88867ff9-6774-405d-b483-04d5790dda5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.785580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79b72cd-0e76-4836-b1a4-9c09aea1252f admin admin] RBAC: Authorizing `identity:create_grant(project_id=836a885e5c97498db1721adb1a22a05f, user_id=c8dcc087badb43b7a18d294a6e316fe0, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.786918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88867ff9-6774-405d-b483-04d5790dda5f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.788256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79b72cd-0e76-4836-b1a4-9c09aea1252f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.797634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 796/1595] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.798506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88867ff9-6774-405d-b483-04d5790dda5f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.802981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 800/1596] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:50 2022] PUT /identity/v3/projects/836a885e5c97498db1721adb1a22a05f/users/c8dcc087badb43b7a18d294a6e316fe0/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:50.803603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79b72cd-0e76-4836-b1a4-9c09aea1252f admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.809273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88867ff9-6774-405d-b483-04d5790dda5f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.866303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1601bea-1e7b-4ca4-beb3-1a330d6486f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.866921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1601bea-1e7b-4ca4-beb3-1a330d6486f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.867121 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79b72cd-0e76-4836-b1a4-9c09aea1252f admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.867253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1601bea-1e7b-4ca4-beb3-1a330d6486f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.867583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1601bea-1e7b-4ca4-beb3-1a330d6486f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.868236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1601bea-1e7b-4ca4-beb3-1a330d6486f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.869855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1601bea-1e7b-4ca4-beb3-1a330d6486f0 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.871779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b70185d-1308-4c24-ab24-29d74854b0b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.872446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b70185d-1308-4c24-ab24-29d74854b0b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.872727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b70185d-1308-4c24-ab24-29d74854b0b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.873078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b70185d-1308-4c24-ab24-29d74854b0b0 admin admin] PATH_INFO: `/v3/projects/6c27cf51a0b3474d84d16e6f53cf938b/users/ffb3221fa4bd48b7af9359b9d59be5ca/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.879965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 797/1597] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:50.880805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1601bea-1e7b-4ca4-beb3-1a330d6486f0 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:50.890211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b70185d-1308-4c24-ab24-29d74854b0b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c27cf51a0b3474d84d16e6f53cf938b, user_id=ffb3221fa4bd48b7af9359b9d59be5ca, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:50.893054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b70185d-1308-4c24-ab24-29d74854b0b0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:50.907267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 801/1598] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:50 2022] PUT /identity/v3/projects/6c27cf51a0b3474d84d16e6f53cf938b/users/ffb3221fa4bd48b7af9359b9d59be5ca/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:50.908606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a138d6-224d-4920-ac0c-83be00022ffe None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.908890 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a138d6-224d-4920-ac0c-83be00022ffe None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.909162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a138d6-224d-4920-ac0c-83be00022ffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:50.936978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-40a138d6-224d-4920-ac0c-83be00022ffe None None] MFA Rules not processed for user `7c4f9efdfd2d498d8f76360bb17fd021`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:50.937187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1601bea-1e7b-4ca4-beb3-1a330d6486f0 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:50.938231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-40a138d6-224d-4920-ac0c-83be00022ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:50.985735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6de589-1cc5-4452-acaf-066ff2215acd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:50.986381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6de589-1cc5-4452-acaf-066ff2215acd service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:50.986833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6de589-1cc5-4452-acaf-066ff2215acd service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:50.986990 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6de589-1cc5-4452-acaf-066ff2215acd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.002058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 802/1599] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:50 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:51.002943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a138d6-224d-4920-ac0c-83be00022ffe None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.040139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:51.046157 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:51.057173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6de589-1cc5-4452-acaf-066ff2215acd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.058616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6de589-1cc5-4452-acaf-066ff2215acd service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.069011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 798/1600] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:51.069833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6de589-1cc5-4452-acaf-066ff2215acd service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.072819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a138d6-224d-4920-ac0c-83be00022ffe None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.077735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970f4d8f-c6a6-4922-af50-d81c7dc52dba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.078504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970f4d8f-c6a6-4922-af50-d81c7dc52dba admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.078881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970f4d8f-c6a6-4922-af50-d81c7dc52dba admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.078998 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6de589-1cc5-4452-acaf-066ff2215acd service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.079512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970f4d8f-c6a6-4922-af50-d81c7dc52dba admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.080628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970f4d8f-c6a6-4922-af50-d81c7dc52dba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.082621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970f4d8f-c6a6-4922-af50-d81c7dc52dba admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.086288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14c3b0a-c7db-4dd9-911c-990cc257802e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.087267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14c3b0a-c7db-4dd9-911c-990cc257802e admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.090633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14c3b0a-c7db-4dd9-911c-990cc257802e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.091252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14c3b0a-c7db-4dd9-911c-990cc257802e admin admin] PATH_INFO: `/v3/projects/8b934b7f515d446786fc24961906d672/users/29157c390595428a9f3803ef341f1e29/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.093243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 803/1601] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:51 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:51.095170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970f4d8f-c6a6-4922-af50-d81c7dc52dba admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.107212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970f4d8f-c6a6-4922-af50-d81c7dc52dba admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.111936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2142e108-d0c1-4b32-92a9-87682bf9f5d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.112330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14c3b0a-c7db-4dd9-911c-990cc257802e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b934b7f515d446786fc24961906d672, user_id=29157c390595428a9f3803ef341f1e29, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.112547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2142e108-d0c1-4b32-92a9-87682bf9f5d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.112824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2142e108-d0c1-4b32-92a9-87682bf9f5d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.113095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2142e108-d0c1-4b32-92a9-87682bf9f5d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.113767 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2142e108-d0c1-4b32-92a9-87682bf9f5d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.115420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2142e108-d0c1-4b32-92a9-87682bf9f5d7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.115529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14c3b0a-c7db-4dd9-911c-990cc257802e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.126733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 804/1602] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:51 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:51.128097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79184be-738e-4a03-b47e-e3a0bd18f75e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.128385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79184be-738e-4a03-b47e-e3a0bd18f75e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.128661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79184be-738e-4a03-b47e-e3a0bd18f75e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.136380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 799/1603] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:51 2022] PUT /identity/v3/projects/8b934b7f515d446786fc24961906d672/users/29157c390595428a9f3803ef341f1e29/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:51.138296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14c3b0a-c7db-4dd9-911c-990cc257802e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.163196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a79184be-738e-4a03-b47e-e3a0bd18f75e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:51.164434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a79184be-738e-4a03-b47e-e3a0bd18f75e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:51.214284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14c3b0a-c7db-4dd9-911c-990cc257802e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.220898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7038a433-20ff-4d3e-a90d-5c8412b3c4f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.221822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7038a433-20ff-4d3e-a90d-5c8412b3c4f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.222065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7038a433-20ff-4d3e-a90d-5c8412b3c4f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.222442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7038a433-20ff-4d3e-a90d-5c8412b3c4f4 admin admin] PATH_INFO: `/v3/projects/4cb02cfe92f1458cb1c02f386c680c82/users/a82232d505b4461990deeb89f8b25cb6/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.240569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 805/1604] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:51.240569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c54b21-d3b7-4f24-b36a-e96a957938c8 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.240569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c54b21-d3b7-4f24-b36a-e96a957938c8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.240569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c54b21-d3b7-4f24-b36a-e96a957938c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.254807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7038a433-20ff-4d3e-a90d-5c8412b3c4f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4cb02cfe92f1458cb1c02f386c680c82, user_id=a82232d505b4461990deeb89f8b25cb6, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.258079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7038a433-20ff-4d3e-a90d-5c8412b3c4f4 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.275050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 800/1605] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:51 2022] PUT /identity/v3/projects/4cb02cfe92f1458cb1c02f386c680c82/users/a82232d505b4461990deeb89f8b25cb6/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:51.276026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-52c54b21-d3b7-4f24-b36a-e96a957938c8 None None] MFA Rules not processed for user `ffb3221fa4bd48b7af9359b9d59be5ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:51.276149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7038a433-20ff-4d3e-a90d-5c8412b3c4f4 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.277429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-52c54b21-d3b7-4f24-b36a-e96a957938c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:51.344344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7038a433-20ff-4d3e-a90d-5c8412b3c4f4 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.350508 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4958e5-8068-438d-ba98-f57ae7363b38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.351121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4958e5-8068-438d-ba98-f57ae7363b38 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.351399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4958e5-8068-438d-ba98-f57ae7363b38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.351668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4958e5-8068-438d-ba98-f57ae7363b38 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.352300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4958e5-8068-438d-ba98-f57ae7363b38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.354246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4958e5-8068-438d-ba98-f57ae7363b38 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.354390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 806/1606] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:51.355542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c54b21-d3b7-4f24-b36a-e96a957938c8 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.369388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c54b21-d3b7-4f24-b36a-e96a957938c8 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.378632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb2ce49-ed86-45ac-97e8-1a43370ff216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.379581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb2ce49-ed86-45ac-97e8-1a43370ff216 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.379884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb2ce49-ed86-45ac-97e8-1a43370ff216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.380154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb2ce49-ed86-45ac-97e8-1a43370ff216 admin admin] PATH_INFO: `/v3/projects/3cbf0c034dd54a8d856d9bc6fb485b45/users/515e0f8a3a234deba5f909271b303abb/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.395245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 801/1607] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/projects => generated 377 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:51.396202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4958e5-8068-438d-ba98-f57ae7363b38 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.402772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb2ce49-ed86-45ac-97e8-1a43370ff216 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cbf0c034dd54a8d856d9bc6fb485b45, user_id=515e0f8a3a234deba5f909271b303abb, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.406038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb2ce49-ed86-45ac-97e8-1a43370ff216 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.429777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 807/1608] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:51 2022] PUT /identity/v3/projects/3cbf0c034dd54a8d856d9bc6fb485b45/users/515e0f8a3a234deba5f909271b303abb/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:51.431409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c050febb-e649-4ec4-9849-39c240a89af6 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.431756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c050febb-e649-4ec4-9849-39c240a89af6 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.432085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c050febb-e649-4ec4-9849-39c240a89af6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.448782 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:51.463067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c050febb-e649-4ec4-9849-39c240a89af6 None None] MFA Rules not processed for user `29157c390595428a9f3803ef341f1e29`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:51.464789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c050febb-e649-4ec4-9849-39c240a89af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:51.482560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4958e5-8068-438d-ba98-f57ae7363b38 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.483292 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:51.555430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fd3cbf-97cc-41b8-8787-64377e02fd40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.556216 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd3cbf-97cc-41b8-8787-64377e02fd40 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.556660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 808/1609] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:51.556878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd3cbf-97cc-41b8-8787-64377e02fd40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.557247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd3cbf-97cc-41b8-8787-64377e02fd40 admin admin] PATH_INFO: `/v3/projects/07f8e7ebb2814be392842c2e17d55fca/users/97189d5c87c94be2bef7cff79c471c7d/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.557364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c050febb-e649-4ec4-9849-39c240a89af6 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.558768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c050febb-e649-4ec4-9849-39c240a89af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:51.575925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fd3cbf-97cc-41b8-8787-64377e02fd40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07f8e7ebb2814be392842c2e17d55fca, user_id=97189d5c87c94be2bef7cff79c471c7d, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.578620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fd3cbf-97cc-41b8-8787-64377e02fd40 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.595877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 802/1610] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:51 2022] PUT /identity/v3/projects/07f8e7ebb2814be392842c2e17d55fca/users/97189d5c87c94be2bef7cff79c471c7d/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:51.597260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d893df-788b-4598-b676-a121e9649db7 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.597558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d893df-788b-4598-b676-a121e9649db7 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.597832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d893df-788b-4598-b676-a121e9649db7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.629360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-72d893df-788b-4598-b676-a121e9649db7 None None] MFA Rules not processed for user `a82232d505b4461990deeb89f8b25cb6`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:51.630843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-72d893df-788b-4598-b676-a121e9649db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:51.634427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c050febb-e649-4ec4-9849-39c240a89af6 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.700740 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ecf76e-68a5-4842-9990-f661a86d16a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.701379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ecf76e-68a5-4842-9990-f661a86d16a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.701668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ecf76e-68a5-4842-9990-f661a86d16a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.701937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ecf76e-68a5-4842-9990-f661a86d16a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.705867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 803/1611] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:51.707167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d893df-788b-4598-b676-a121e9649db7 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.772542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ecf76e-68a5-4842-9990-f661a86d16a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.774513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ecf76e-68a5-4842-9990-f661a86d16a6 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.777269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:51.782050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d893df-788b-4598-b676-a121e9649db7 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.786931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 809/1612] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:51.788228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebed3096-49f6-4b75-8caf-a7a4779e0740 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.788506 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebed3096-49f6-4b75-8caf-a7a4779e0740 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.788771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebed3096-49f6-4b75-8caf-a7a4779e0740 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.792161 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 810/1613] 10.222.0.16 () {54 vars in 962 bytes} [Tue Sep 13 15:29:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:29:51.792242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c104419-a623-46b4-8ddb-b6b63b1ca0be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.792242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c104419-a623-46b4-8ddb-b6b63b1ca0be admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.795381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b37c26d-7dc9-4dd6-bd92-cf25033e627e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.795922 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b37c26d-7dc9-4dd6-bd92-cf25033e627e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.796424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b37c26d-7dc9-4dd6-bd92-cf25033e627e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.797946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c104419-a623-46b4-8ddb-b6b63b1ca0be admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.798304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c104419-a623-46b4-8ddb-b6b63b1ca0be admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.799027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c104419-a623-46b4-8ddb-b6b63b1ca0be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.801037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c104419-a623-46b4-8ddb-b6b63b1ca0be admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.819068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4b37c26d-7dc9-4dd6-bd92-cf25033e627e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:51.820394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4b37c26d-7dc9-4dd6-bd92-cf25033e627e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:51.828207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 804/1614] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/users => generated 295 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:51.829304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c104419-a623-46b4-8ddb-b6b63b1ca0be admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.829741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 811/1615] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:51.831065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5365760-1c86-4574-aa68-a8490ccddc94 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.831347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5365760-1c86-4574-aa68-a8490ccddc94 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.831617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5365760-1c86-4574-aa68-a8490ccddc94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.840970 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c104419-a623-46b4-8ddb-b6b63b1ca0be admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.845811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72e4b61-8064-443e-9d1a-1ebbecbbda5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.846460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72e4b61-8064-443e-9d1a-1ebbecbbda5f admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.846776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72e4b61-8064-443e-9d1a-1ebbecbbda5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.847083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72e4b61-8064-443e-9d1a-1ebbecbbda5f admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.847829 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72e4b61-8064-443e-9d1a-1ebbecbbda5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.850517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72e4b61-8064-443e-9d1a-1ebbecbbda5f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:51.862585 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-f5365760-1c86-4574-aa68-a8490ccddc94 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:51.865607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f5365760-1c86-4574-aa68-a8490ccddc94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:51.879892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 812/1616] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:51.881209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5365760-1c86-4574-aa68-a8490ccddc94 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:51.884646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 805/1617] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/projects => generated 373 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:51.886219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e6f0f-62e1-438a-95ea-a7ecd2fb54d2 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.886515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e6f0f-62e1-438a-95ea-a7ecd2fb54d2 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.886783 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e6f0f-62e1-438a-95ea-a7ecd2fb54d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.890961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:51.897645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5365760-1c86-4574-aa68-a8490ccddc94 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:51.951622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-109e6f0f-62e1-438a-95ea-a7ecd2fb54d2 None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:51.953181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-109e6f0f-62e1-438a-95ea-a7ecd2fb54d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:51.972281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bf0625-2aaf-482c-928c-514e014174fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:51.973054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bf0625-2aaf-482c-928c-514e014174fa admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:51.973431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bf0625-2aaf-482c-928c-514e014174fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:51.974263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bf0625-2aaf-482c-928c-514e014174fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:51.975074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bf0625-2aaf-482c-928c-514e014174fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:51.977497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bf0625-2aaf-482c-928c-514e014174fa admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.022603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 813/1618] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/projects => generated 385 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:52.027370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bf0625-2aaf-482c-928c-514e014174fa admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.052486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 806/1619] 10.222.0.16 () {58 vars in 1059 bytes} [Tue Sep 13 15:29:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:52.053707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109e6f0f-62e1-438a-95ea-a7ecd2fb54d2 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.120853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109e6f0f-62e1-438a-95ea-a7ecd2fb54d2 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.183623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bf0625-2aaf-482c-928c-514e014174fa admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.197447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ad030d-39dd-4629-86cb-bbe08a414e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.198088 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ad030d-39dd-4629-86cb-bbe08a414e24 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.198395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ad030d-39dd-4629-86cb-bbe08a414e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.198667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ad030d-39dd-4629-86cb-bbe08a414e24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.199323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ad030d-39dd-4629-86cb-bbe08a414e24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.200771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f82b83c-6b8b-4916-97cc-e182571a9232 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.201018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ad030d-39dd-4629-86cb-bbe08a414e24 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.201396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82b83c-6b8b-4916-97cc-e182571a9232 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.201672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82b83c-6b8b-4916-97cc-e182571a9232 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.201947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82b83c-6b8b-4916-97cc-e182571a9232 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.213046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f82b83c-6b8b-4916-97cc-e182571a9232 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.214634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f82b83c-6b8b-4916-97cc-e182571a9232 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.220766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:52.224252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 814/1620] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:52 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:52.227405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 807/1621] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:52.230564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f82b83c-6b8b-4916-97cc-e182571a9232 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.230817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ad030d-39dd-4629-86cb-bbe08a414e24 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.243212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f82b83c-6b8b-4916-97cc-e182571a9232 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.250247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24569e7-b415-4ca3-9ee9-b85e15047b1a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.250929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24569e7-b415-4ca3-9ee9-b85e15047b1a service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.251381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24569e7-b415-4ca3-9ee9-b85e15047b1a service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.251564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24569e7-b415-4ca3-9ee9-b85e15047b1a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.253683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ad030d-39dd-4629-86cb-bbe08a414e24 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.268129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6ddf32-a619-48e8-95a3-60fff26915d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.268319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24569e7-b415-4ca3-9ee9-b85e15047b1a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.270132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6ddf32-a619-48e8-95a3-60fff26915d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.270261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24569e7-b415-4ca3-9ee9-b85e15047b1a service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.270476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6ddf32-a619-48e8-95a3-60fff26915d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.272083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6ddf32-a619-48e8-95a3-60fff26915d3 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.273138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6ddf32-a619-48e8-95a3-60fff26915d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.278982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6ddf32-a619-48e8-95a3-60fff26915d3 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.289977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 808/1622] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:52.290924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24569e7-b415-4ca3-9ee9-b85e15047b1a service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.302093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24569e7-b415-4ca3-9ee9-b85e15047b1a service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.308331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4514fbf0-d7ae-48ef-9e8e-0e963e280356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.309357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4514fbf0-d7ae-48ef-9e8e-0e963e280356 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.309854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4514fbf0-d7ae-48ef-9e8e-0e963e280356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.310204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4514fbf0-d7ae-48ef-9e8e-0e963e280356 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.311035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4514fbf0-d7ae-48ef-9e8e-0e963e280356 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.313211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4514fbf0-d7ae-48ef-9e8e-0e963e280356 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.325705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 815/1623] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:52 2022] POST /identity/v3/users => generated 293 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:52.325705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6ddf32-a619-48e8-95a3-60fff26915d3 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.351676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 809/1624] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:52 2022] POST /identity/v3/users => generated 299 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:52.352846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4514fbf0-d7ae-48ef-9e8e-0e963e280356 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.368276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4514fbf0-d7ae-48ef-9e8e-0e963e280356 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.374172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84add17c-2cd6-48a7-b4cc-03508a3baeba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.375340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84add17c-2cd6-48a7-b4cc-03508a3baeba admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.375815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84add17c-2cd6-48a7-b4cc-03508a3baeba admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.376360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84add17c-2cd6-48a7-b4cc-03508a3baeba admin admin] PATH_INFO: `/v3/projects/09c41d0a733a421280da4085ac2b3825/users/416ef1fbdca54e80ac09b6648cda5642/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.394801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84add17c-2cd6-48a7-b4cc-03508a3baeba admin admin] RBAC: Authorizing `identity:create_grant(project_id=09c41d0a733a421280da4085ac2b3825, user_id=416ef1fbdca54e80ac09b6648cda5642, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.400921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84add17c-2cd6-48a7-b4cc-03508a3baeba admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.432761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 810/1625] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:52 2022] PUT /identity/v3/projects/09c41d0a733a421280da4085ac2b3825/users/416ef1fbdca54e80ac09b6648cda5642/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:52.436156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84add17c-2cd6-48a7-b4cc-03508a3baeba admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.456810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:52.463279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6ddf32-a619-48e8-95a3-60fff26915d3 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.532486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84add17c-2cd6-48a7-b4cc-03508a3baeba admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.540687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b660303b-4d2a-45b6-92c8-ccb97aab47b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.541501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b660303b-4d2a-45b6-92c8-ccb97aab47b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.542796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b660303b-4d2a-45b6-92c8-ccb97aab47b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.543147 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b660303b-4d2a-45b6-92c8-ccb97aab47b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.544149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b660303b-4d2a-45b6-92c8-ccb97aab47b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.545938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b660303b-4d2a-45b6-92c8-ccb97aab47b6 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.554649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740c3156-d4bd-479b-9c10-f5b4f30d5f05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.555726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740c3156-d4bd-479b-9c10-f5b4f30d5f05 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.556227 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740c3156-d4bd-479b-9c10-f5b4f30d5f05 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.556808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740c3156-d4bd-479b-9c10-f5b4f30d5f05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.561314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 811/1626] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:52 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:52.562371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b660303b-4d2a-45b6-92c8-ccb97aab47b6 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.574255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b660303b-4d2a-45b6-92c8-ccb97aab47b6 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.582705 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a4e98b-5728-4354-877c-a9229c26d672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.584188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a4e98b-5728-4354-877c-a9229c26d672 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.584624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a4e98b-5728-4354-877c-a9229c26d672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.585136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a4e98b-5728-4354-877c-a9229c26d672 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.585914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a4e98b-5728-4354-877c-a9229c26d672 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.588640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a4e98b-5728-4354-877c-a9229c26d672 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.605234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 812/1627] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:52 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:52.606345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a4e98b-5728-4354-877c-a9229c26d672 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.620610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a4e98b-5728-4354-877c-a9229c26d672 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.633323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94450e24-7b4a-4c74-8f36-1356ac4e5237 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.636272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94450e24-7b4a-4c74-8f36-1356ac4e5237 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.636748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94450e24-7b4a-4c74-8f36-1356ac4e5237 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.637670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94450e24-7b4a-4c74-8f36-1356ac4e5237 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.640326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94450e24-7b4a-4c74-8f36-1356ac4e5237 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.643022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740c3156-d4bd-479b-9c10-f5b4f30d5f05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.643759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94450e24-7b4a-4c74-8f36-1356ac4e5237 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.645770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740c3156-d4bd-479b-9c10-f5b4f30d5f05 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.657060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 816/1628] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:52 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:52.658335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740c3156-d4bd-479b-9c10-f5b4f30d5f05 service nova] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.671318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740c3156-d4bd-479b-9c10-f5b4f30d5f05 service nova] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.673300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 813/1629] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:52 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:52.675791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94450e24-7b4a-4c74-8f36-1356ac4e5237 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.677524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326a8fc5-87c3-4dd3-b683-72d12668e4fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.678459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326a8fc5-87c3-4dd3-b683-72d12668e4fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.678924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326a8fc5-87c3-4dd3-b683-72d12668e4fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.679367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326a8fc5-87c3-4dd3-b683-72d12668e4fe admin admin] PATH_INFO: `/v3/projects/e5a6119b7301452bb4181c5b9481c7a9/users/25dc9fc5ed4040f88edb5b56c80bd5f7/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.699096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326a8fc5-87c3-4dd3-b683-72d12668e4fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5a6119b7301452bb4181c5b9481c7a9, user_id=25dc9fc5ed4040f88edb5b56c80bd5f7, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.702073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326a8fc5-87c3-4dd3-b683-72d12668e4fe admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.702354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94450e24-7b4a-4c74-8f36-1356ac4e5237 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.714870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50d36c1-c6d2-40cc-bca9-1ba4c02a975a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.716981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50d36c1-c6d2-40cc-bca9-1ba4c02a975a admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.717656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50d36c1-c6d2-40cc-bca9-1ba4c02a975a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.718477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50d36c1-c6d2-40cc-bca9-1ba4c02a975a admin admin] PATH_INFO: `/v3/projects/a0e48f36f5434187b7b14718a5ffe290/users/abe3610aa61448c0bc2945884b4c29c5/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.725985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 817/1630] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:52 2022] PUT /identity/v3/projects/e5a6119b7301452bb4181c5b9481c7a9/users/25dc9fc5ed4040f88edb5b56c80bd5f7/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:52.727911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326a8fc5-87c3-4dd3-b683-72d12668e4fe admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.770420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50d36c1-c6d2-40cc-bca9-1ba4c02a975a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0e48f36f5434187b7b14718a5ffe290, user_id=abe3610aa61448c0bc2945884b4c29c5, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.775566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50d36c1-c6d2-40cc-bca9-1ba4c02a975a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.807111 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 814/1631] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:52 2022] PUT /identity/v3/projects/a0e48f36f5434187b7b14718a5ffe290/users/abe3610aa61448c0bc2945884b4c29c5/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:52.808245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50d36c1-c6d2-40cc-bca9-1ba4c02a975a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.846355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326a8fc5-87c3-4dd3-b683-72d12668e4fe admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.875104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:52.930884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9342391-8f8e-4a96-8372-c1033ac5aaad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.931676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9342391-8f8e-4a96-8372-c1033ac5aaad admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.931958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9342391-8f8e-4a96-8372-c1033ac5aaad admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.932228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9342391-8f8e-4a96-8372-c1033ac5aaad admin admin] PATH_INFO: `/v3/projects/09c41d0a733a421280da4085ac2b3825/users/416ef1fbdca54e80ac09b6648cda5642/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.935002 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:52.936219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9342391-8f8e-4a96-8372-c1033ac5aaad admin admin] RBAC: Authorizing `identity:create_grant(project_id=09c41d0a733a421280da4085ac2b3825, user_id=416ef1fbdca54e80ac09b6648cda5642, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.939624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9342391-8f8e-4a96-8372-c1033ac5aaad admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.941671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50d36c1-c6d2-40cc-bca9-1ba4c02a975a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:52.948704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee105c43-97aa-40d4-9ba0-00c6605fefb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:52.950357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee105c43-97aa-40d4-9ba0-00c6605fefb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.951082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee105c43-97aa-40d4-9ba0-00c6605fefb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.951758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee105c43-97aa-40d4-9ba0-00c6605fefb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:52.952568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee105c43-97aa-40d4-9ba0-00c6605fefb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:52.954989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee105c43-97aa-40d4-9ba0-00c6605fefb7 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:52.965672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 818/1632] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:52 2022] PUT /identity/v3/projects/09c41d0a733a421280da4085ac2b3825/users/416ef1fbdca54e80ac09b6648cda5642/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:52.967010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9342391-8f8e-4a96-8372-c1033ac5aaad admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:52.970857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 815/1633] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:52 2022] GET /identity/v3/roles => generated 1420 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:52.970857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1df7533-78e2-4201-bc8a-229eade5041e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:52.970857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1df7533-78e2-4201-bc8a-229eade5041e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:52.970857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1df7533-78e2-4201-bc8a-229eade5041e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:53.004469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-a1df7533-78e2-4201-bc8a-229eade5041e None None] MFA Rules not processed for user `416ef1fbdca54e80ac09b6648cda5642`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:53.006228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a1df7533-78e2-4201-bc8a-229eade5041e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:53.050794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9342391-8f8e-4a96-8372-c1033ac5aaad admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:53.057461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb9735f-8a2a-41a8-8bbf-6f3308727b60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:53.058666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb9735f-8a2a-41a8-8bbf-6f3308727b60 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:53.059291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb9735f-8a2a-41a8-8bbf-6f3308727b60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:53.059822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb9735f-8a2a-41a8-8bbf-6f3308727b60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:53.060638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb9735f-8a2a-41a8-8bbf-6f3308727b60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:53.062668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb9735f-8a2a-41a8-8bbf-6f3308727b60 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:53.076404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 819/1634] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:52 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:53.077357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb9735f-8a2a-41a8-8bbf-6f3308727b60 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:53.093076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb9735f-8a2a-41a8-8bbf-6f3308727b60 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:53.108403 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb911e65-bd45-464b-9834-40be052a1cb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:53.110068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 816/1635] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:52 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:53.111435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb911e65-bd45-464b-9834-40be052a1cb5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:53.112155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb911e65-bd45-464b-9834-40be052a1cb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:53.112427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1df7533-78e2-4201-bc8a-229eade5041e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:53.112905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb911e65-bd45-464b-9834-40be052a1cb5 admin admin] PATH_INFO: `/v3/projects/e5a6119b7301452bb4181c5b9481c7a9/users/25dc9fc5ed4040f88edb5b56c80bd5f7/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:53.119415 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb911e65-bd45-464b-9834-40be052a1cb5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5a6119b7301452bb4181c5b9481c7a9, user_id=25dc9fc5ed4040f88edb5b56c80bd5f7, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:53.124250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb911e65-bd45-464b-9834-40be052a1cb5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:53.140818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1df7533-78e2-4201-bc8a-229eade5041e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:53.162050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 820/1636] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:53 2022] PUT /identity/v3/projects/e5a6119b7301452bb4181c5b9481c7a9/users/25dc9fc5ed4040f88edb5b56c80bd5f7/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:53.164710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb911e65-bd45-464b-9834-40be052a1cb5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:53.293749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff57235b-7fb8-4f22-9b87-907d3ba8d8d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:53.294776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff57235b-7fb8-4f22-9b87-907d3ba8d8d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:53.296257 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff57235b-7fb8-4f22-9b87-907d3ba8d8d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:53.297942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff57235b-7fb8-4f22-9b87-907d3ba8d8d6 admin admin] PATH_INFO: `/v3/projects/a0e48f36f5434187b7b14718a5ffe290/users/abe3610aa61448c0bc2945884b4c29c5/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:53.302584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff57235b-7fb8-4f22-9b87-907d3ba8d8d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0e48f36f5434187b7b14718a5ffe290, user_id=abe3610aa61448c0bc2945884b4c29c5, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:53.307709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb911e65-bd45-464b-9834-40be052a1cb5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:53.309669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff57235b-7fb8-4f22-9b87-907d3ba8d8d6 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:53.315843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fa08b9-5aee-4477-bdd4-332c51f71d83 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:53.317328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fa08b9-5aee-4477-bdd4-332c51f71d83 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:53.317863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fa08b9-5aee-4477-bdd4-332c51f71d83 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:53.318860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fa08b9-5aee-4477-bdd4-332c51f71d83 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:53.350245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 817/1637] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:53 2022] PUT /identity/v3/projects/a0e48f36f5434187b7b14718a5ffe290/users/abe3610aa61448c0bc2945884b4c29c5/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:53.350245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc4ac29-fea9-4da3-a659-4030fde6460d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:53.350245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc4ac29-fea9-4da3-a659-4030fde6460d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:53.350245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc4ac29-fea9-4da3-a659-4030fde6460d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:53.420611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-ecc4ac29-fea9-4da3-a659-4030fde6460d None None] MFA Rules not processed for user `25dc9fc5ed4040f88edb5b56c80bd5f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:53.422270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ecc4ac29-fea9-4da3-a659-4030fde6460d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:53.450524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:53.477140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fa08b9-5aee-4477-bdd4-332c51f71d83 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:53.478752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fa08b9-5aee-4477-bdd4-332c51f71d83 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:53.536047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 821/1638] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:53 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:53.540439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cf2dd4-5efe-4ca2-a583-1a1338084eae None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:53.542901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cf2dd4-5efe-4ca2-a583-1a1338084eae None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:53.543335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cf2dd4-5efe-4ca2-a583-1a1338084eae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:53.594388 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:53.619617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 818/1639] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:53 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:53.638390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc4ac29-fea9-4da3-a659-4030fde6460d None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:53.662945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-79cf2dd4-5efe-4ca2-a583-1a1338084eae None None] MFA Rules not processed for user `abe3610aa61448c0bc2945884b4c29c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:53.673411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-79cf2dd4-5efe-4ca2-a583-1a1338084eae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:53.784016 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc4ac29-fea9-4da3-a659-4030fde6460d None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:53.807398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5fc6bd-1c45-40a4-8f7a-7bc2d2bc4da4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:53.808472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5fc6bd-1c45-40a4-8f7a-7bc2d2bc4da4 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:53.808750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5fc6bd-1c45-40a4-8f7a-7bc2d2bc4da4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:53.809175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5fc6bd-1c45-40a4-8f7a-7bc2d2bc4da4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:53.813493 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 822/1640] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:53 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:53.814985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57076041-0619-4c9f-9c3d-509ab50b7935 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:53.815275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57076041-0619-4c9f-9c3d-509ab50b7935 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:53.815550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57076041-0619-4c9f-9c3d-509ab50b7935 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:53.821398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 823/1641] 10.222.0.16 () {54 vars in 962 bytes} [Tue Sep 13 15:29:53 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:29:53.822334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57076041-0619-4c9f-9c3d-509ab50b7935 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:53.835354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57076041-0619-4c9f-9c3d-509ab50b7935 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:53.838064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5fc6bd-1c45-40a4-8f7a-7bc2d2bc4da4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:53.839050 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ebff1f-dac8-4973-b904-ddc3246ed212 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:53.839668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ebff1f-dac8-4973-b904-ddc3246ed212 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:53.840530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ebff1f-dac8-4973-b904-ddc3246ed212 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:53.840530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ebff1f-dac8-4973-b904-ddc3246ed212 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:53.841389 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5fc6bd-1c45-40a4-8f7a-7bc2d2bc4da4 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:53.873254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 819/1642] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:53 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:53.878047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ee6a7b-7678-4b0a-b9da-0ae9abadd4f4 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:53.878375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ee6a7b-7678-4b0a-b9da-0ae9abadd4f4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:53.879320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ee6a7b-7678-4b0a-b9da-0ae9abadd4f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:54.011878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ebff1f-dac8-4973-b904-ddc3246ed212 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:54.011878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ebff1f-dac8-4973-b904-ddc3246ed212 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:54.012719 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-51ee6a7b-7678-4b0a-b9da-0ae9abadd4f4 None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:54.012719 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-51ee6a7b-7678-4b0a-b9da-0ae9abadd4f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:54.013124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 824/1643] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:53 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:54.013124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ebff1f-dac8-4973-b904-ddc3246ed212 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:54.013124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ebff1f-dac8-4973-b904-ddc3246ed212 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:54.021256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e617431-e7a7-4766-b041-d36913b11cf0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:54.023754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e617431-e7a7-4766-b041-d36913b11cf0 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:54.024062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e617431-e7a7-4766-b041-d36913b11cf0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:54.024598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e617431-e7a7-4766-b041-d36913b11cf0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:54.051212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e617431-e7a7-4766-b041-d36913b11cf0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:54.052631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e617431-e7a7-4766-b041-d36913b11cf0 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:54.083551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 825/1644] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:53 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:54.147533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e617431-e7a7-4766-b041-d36913b11cf0 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:54.187390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 820/1645] 10.222.0.16 () {58 vars in 1059 bytes} [Tue Sep 13 15:29:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:54.194142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce20feae-aa0e-412b-ab17-12760f99df33 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:54.194963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce20feae-aa0e-412b-ab17-12760f99df33 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:54.195254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce20feae-aa0e-412b-ab17-12760f99df33 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:54.196704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 821/1646] 10.222.0.16 () {54 vars in 912 bytes} [Tue Sep 13 15:29:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:29:54.202560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f864e34-9397-4bb5-8cdf-83c4023c2b76 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:54.202843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f864e34-9397-4bb5-8cdf-83c4023c2b76 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:54.203118 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f864e34-9397-4bb5-8cdf-83c4023c2b76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:54.256415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-9f864e34-9397-4bb5-8cdf-83c4023c2b76 None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:54.257770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9f864e34-9397-4bb5-8cdf-83c4023c2b76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:54.294739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 822/1647] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:29:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:29:54.336475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f864e34-9397-4bb5-8cdf-83c4023c2b76 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:54.351773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f864e34-9397-4bb5-8cdf-83c4023c2b76 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:54.359412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e617431-e7a7-4766-b041-d36913b11cf0 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:54.371668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f31cb36-3ee3-483a-b4e0-595bed02eea8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:54.373391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31cb36-3ee3-483a-b4e0-595bed02eea8 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:54.374955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:54.375116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31cb36-3ee3-483a-b4e0-595bed02eea8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:54.375275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31cb36-3ee3-483a-b4e0-595bed02eea8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:54.375957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:54.376240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:54.376512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] PATH_INFO: `/v3/users/a82232d505b4461990deeb89f8b25cb6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:54.377923 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a82232d505b4461990deeb89f8b25cb6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:54.379040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: WARNING py.warnings [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:29:54.379040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: warnings.warn(msg) Sep 13 15:29:54.379040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: Sep 13 15:29:54.380210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:54.387882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:54.408220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f31cb36-3ee3-483a-b4e0-595bed02eea8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:54.412314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f31cb36-3ee3-483a-b4e0-595bed02eea8 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:54.441323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 823/1648] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:54.511442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f31cb36-3ee3-483a-b4e0-595bed02eea8 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:54.521295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f31cb36-3ee3-483a-b4e0-595bed02eea8 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:54.534068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd7faf2-7ebc-488b-885e-8f43a02ddcc2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:54.534868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd7faf2-7ebc-488b-885e-8f43a02ddcc2 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:54.535362 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd7faf2-7ebc-488b-885e-8f43a02ddcc2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:54.535732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd7faf2-7ebc-488b-885e-8f43a02ddcc2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:54.587749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a82232d505b4461990deeb89f8b25cb6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:54.611424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a82232d505b4461990deeb89f8b25cb6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:54.612461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a82232d505b4461990deeb89f8b25cb6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:54.613335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a82232d505b4461990deeb89f8b25cb6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:54.687858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0af9e36d-d7c1-4083-b83a-8a50047a1e01 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a82232d505b4461990deeb89f8b25cb6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:54.750528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd7faf2-7ebc-488b-885e-8f43a02ddcc2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:54.759098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd7faf2-7ebc-488b-885e-8f43a02ddcc2 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:54.787432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 826/1649] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:29:54 2022] DELETE /identity/v3/users/a82232d505b4461990deeb89f8b25cb6 => generated 0 bytes in 640 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:54.792864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a54aeb-80cb-480f-a85e-97c97f8f5cbe None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:54.793991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a54aeb-80cb-480f-a85e-97c97f8f5cbe None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:54.794481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a54aeb-80cb-480f-a85e-97c97f8f5cbe None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:54.795590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 824/1650] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:54.797086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8456952-5543-4cd5-8ca9-c8bbd986d46d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:54.797375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8456952-5543-4cd5-8ca9-c8bbd986d46d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:54.797645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8456952-5543-4cd5-8ca9-c8bbd986d46d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:54.802650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 827/1651] 10.222.0.16 () {54 vars in 912 bytes} [Tue Sep 13 15:29:54 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:29:54.807573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eb073b-9f80-46db-910b-6e4ebe5b41cb None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:54.808005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eb073b-9f80-46db-910b-6e4ebe5b41cb None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:54.808477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eb073b-9f80-46db-910b-6e4ebe5b41cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:54.846755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e8456952-5543-4cd5-8ca9-c8bbd986d46d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:54.850266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e8456952-5543-4cd5-8ca9-c8bbd986d46d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:54.860916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:54.862996 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-f0eb073b-9f80-46db-910b-6e4ebe5b41cb None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:54.866881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f0eb073b-9f80-46db-910b-6e4ebe5b41cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:55.053863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 825/1652] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:55.121374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8456952-5543-4cd5-8ca9-c8bbd986d46d None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:55.124909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e8456952-5543-4cd5-8ca9-c8bbd986d46d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:55.149709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 828/1653] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:29:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 343 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:55.170686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0eb073b-9f80-46db-910b-6e4ebe5b41cb None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:55.172894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f0eb073b-9f80-46db-910b-6e4ebe5b41cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:55.286363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0eb073b-9f80-46db-910b-6e4ebe5b41cb None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:55.290954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97e5a36-d804-482c-9061-25494089d28f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:55.291679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97e5a36-d804-482c-9061-25494089d28f service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:55.292031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97e5a36-d804-482c-9061-25494089d28f service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:55.292304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97e5a36-d804-482c-9061-25494089d28f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:55.296074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8456952-5543-4cd5-8ca9-c8bbd986d46d None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:55.300844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567c0de0-eaa7-4cce-ada6-660b35c25456 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:55.301812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567c0de0-eaa7-4cce-ada6-660b35c25456 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:55.302126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567c0de0-eaa7-4cce-ada6-660b35c25456 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:55.302402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567c0de0-eaa7-4cce-ada6-660b35c25456 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:55.306463 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97e5a36-d804-482c-9061-25494089d28f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:55.308231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97e5a36-d804-482c-9061-25494089d28f service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:55.324983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567c0de0-eaa7-4cce-ada6-660b35c25456 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:55.324983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567c0de0-eaa7-4cce-ada6-660b35c25456 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:55.332966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 829/1654] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:55.347573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 826/1655] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:55.497471 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97e5a36-d804-482c-9061-25494089d28f service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:55.499025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b97e5a36-d804-482c-9061-25494089d28f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:55.519856 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97e5a36-d804-482c-9061-25494089d28f service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:55.535903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:55.535903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:55.535903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:55.536255 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] PATH_INFO: `/v3/users/ffb3221fa4bd48b7af9359b9d59be5ca` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:55.537380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffb3221fa4bd48b7af9359b9d59be5ca)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:55.539728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:55.556400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:55.664571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffb3221fa4bd48b7af9359b9d59be5ca'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:55.679209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffb3221fa4bd48b7af9359b9d59be5ca'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:55.681401 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffb3221fa4bd48b7af9359b9d59be5ca'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:55.682336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffb3221fa4bd48b7af9359b9d59be5ca'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:55.696662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffb3221fa4bd48b7af9359b9d59be5ca'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:55.727812 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 830/1656] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:29:55 2022] DELETE /identity/v3/users/ffb3221fa4bd48b7af9359b9d59be5ca => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:55.732865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5cbc48-4ec3-4ba3-bc67-d78f4bbfb6aa None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:55.733246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5cbc48-4ec3-4ba3-bc67-d78f4bbfb6aa None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:55.733588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5cbc48-4ec3-4ba3-bc67-d78f4bbfb6aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:55.783107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-ae5cbc48-4ec3-4ba3-bc67-d78f4bbfb6aa None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:55.784460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ae5cbc48-4ec3-4ba3-bc67-d78f4bbfb6aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:55.920533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 827/1657] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:55.962384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:55.965011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:56.074129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb35ed9-9c16-4c59-8480-5e8f6244ee5e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:56.081596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481390d4-d27a-4e0e-a377-b71a8aa00c6d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:56.082539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481390d4-d27a-4e0e-a377-b71a8aa00c6d service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:56.082882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481390d4-d27a-4e0e-a377-b71a8aa00c6d service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:56.083226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481390d4-d27a-4e0e-a377-b71a8aa00c6d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:56.105323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481390d4-d27a-4e0e-a377-b71a8aa00c6d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:56.110939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481390d4-d27a-4e0e-a377-b71a8aa00c6d service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:56.138632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 831/1658] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:29:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:29:57.013562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5cbc48-4ec3-4ba3-bc67-d78f4bbfb6aa None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:57.017031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ae5cbc48-4ec3-4ba3-bc67-d78f4bbfb6aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:57.037857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5cbc48-4ec3-4ba3-bc67-d78f4bbfb6aa None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:57.064617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:57.065476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:57.065789 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:57.066149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] PATH_INFO: `/v3/projects/6c27cf51a0b3474d84d16e6f53cf938b` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:57.070032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c27cf51a0b3474d84d16e6f53cf938b)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:57.071212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: WARNING py.warnings [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 13 15:29:57.071212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: warnings.warn(msg) Sep 13 15:29:57.071212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: Sep 13 15:29:57.072674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:57.225863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:57.227672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c27cf51a0b3474d84d16e6f53cf938b', 'request_id': 'req-0049ed5f-9cdf-45a4-a04c-8266b68c3037'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:57.235540 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c27cf51a0b3474d84d16e6f53cf938b', 'request_id': 'req-0049ed5f-9cdf-45a4-a04c-8266b68c3037'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:57.247211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c27cf51a0b3474d84d16e6f53cf938b', 'request_id': 'req-0049ed5f-9cdf-45a4-a04c-8266b68c3037'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:57.275338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] The token cache is being invalidate because project 6c27cf51a0b3474d84d16e6f53cf938b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:29:57.275705 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:57.276658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:57.278947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 828/1659] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:29:57 2022] DELETE /identity/v3/projects/6c27cf51a0b3474d84d16e6f53cf938b => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:57.284178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a045c665-b53f-475a-9147-79a736bbc7ab None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:57.284689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a045c665-b53f-475a-9147-79a736bbc7ab None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:57.284689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a045c665-b53f-475a-9147-79a736bbc7ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:57.329568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a045c665-b53f-475a-9147-79a736bbc7ab None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:57.340653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a045c665-b53f-475a-9147-79a736bbc7ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:57.393243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:57.395052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:57.530291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 832/1660] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:57.534549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c31cb2e-8732-4186-897a-15191418c612 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:57.534830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c31cb2e-8732-4186-897a-15191418c612 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:57.535901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c31cb2e-8732-4186-897a-15191418c612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:57.605197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8c31cb2e-8732-4186-897a-15191418c612 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:57.611610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8c31cb2e-8732-4186-897a-15191418c612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:57.619585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0049ed5f-9cdf-45a4-a04c-8266b68c3037 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:57.626416 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:57.627428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:57.627829 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:57.628213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] PATH_INFO: `/v3/projects/4cb02cfe92f1458cb1c02f386c680c82` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:57.629544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] RBAC: Authorizing `identity:delete_project(project_id=4cb02cfe92f1458cb1c02f386c680c82)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:57.632139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:57.633971 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 833/1661] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:57.634948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c31cb2e-8732-4186-897a-15191418c612 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:57.656969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c31cb2e-8732-4186-897a-15191418c612 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:57.673103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239abb43-304f-4b3c-835f-2d8f2ccf8d7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:57.673830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239abb43-304f-4b3c-835f-2d8f2ccf8d7a admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:57.674127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239abb43-304f-4b3c-835f-2d8f2ccf8d7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:57.674443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239abb43-304f-4b3c-835f-2d8f2ccf8d7a admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:57.675385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239abb43-304f-4b3c-835f-2d8f2ccf8d7a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:29:57.675730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239abb43-304f-4b3c-835f-2d8f2ccf8d7a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:57.677436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239abb43-304f-4b3c-835f-2d8f2ccf8d7a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:57.712888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 834/1662] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:29:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:29:57.713937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239abb43-304f-4b3c-835f-2d8f2ccf8d7a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:57.742431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239abb43-304f-4b3c-835f-2d8f2ccf8d7a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:57.755033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4806b9-d00a-44df-ab5c-1f7a6811b3da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:57.755680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4806b9-d00a-44df-ab5c-1f7a6811b3da admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:57.758146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4806b9-d00a-44df-ab5c-1f7a6811b3da admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:57.758495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4806b9-d00a-44df-ab5c-1f7a6811b3da admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:57.759196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4806b9-d00a-44df-ab5c-1f7a6811b3da admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:57.767981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4806b9-d00a-44df-ab5c-1f7a6811b3da admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:57.787891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4cb02cfe92f1458cb1c02f386c680c82', 'request_id': 'req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:57.796499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:57.808149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4cb02cfe92f1458cb1c02f386c680c82', 'request_id': 'req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:57.831766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4cb02cfe92f1458cb1c02f386c680c82', 'request_id': 'req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:57.848175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 835/1663] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:57 2022] POST /identity/v3/projects => generated 403 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:57.850503 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4806b9-d00a-44df-ab5c-1f7a6811b3da admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:57.855725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] The token cache is being invalidate because project 4cb02cfe92f1458cb1c02f386c680c82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:29:57.856232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:57.859949 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:57.861712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 829/1664] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:29:57 2022] DELETE /identity/v3/projects/4cb02cfe92f1458cb1c02f386c680c82 => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:29:57.865608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:57.869572 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:57.953864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4806b9-d00a-44df-ab5c-1f7a6811b3da admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:57.959159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d082df9-fda9-41a9-9084-41900e420181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:57.959787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d082df9-fda9-41a9-9084-41900e420181 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:57.960478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d082df9-fda9-41a9-9084-41900e420181 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:57.960743 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d082df9-fda9-41a9-9084-41900e420181 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:57.961387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d082df9-fda9-41a9-9084-41900e420181 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:57.965007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d082df9-fda9-41a9-9084-41900e420181 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:58.001025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 836/1665] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:57 2022] POST /identity/v3/users => generated 302 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:58.002484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d082df9-fda9-41a9-9084-41900e420181 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:58.006055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4d082df9-fda9-41a9-9084-41900e420181 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:58.010283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfbed93-6d93-4d10-97f7-7dc362cc9deb admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:58.025633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:58.026466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:58.026845 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:58.027233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] PATH_INFO: `/v3/users/abe3610aa61448c0bc2945884b4c29c5` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:58.028860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] RBAC: Authorizing `identity:delete_user(user_id=abe3610aa61448c0bc2945884b4c29c5)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:58.031017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:58.047862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:58.061443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d082df9-fda9-41a9-9084-41900e420181 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:58.078584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7171d6d1-0dc1-4cc5-982a-afa08b09881b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:58.079993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7171d6d1-0dc1-4cc5-982a-afa08b09881b admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:58.080412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7171d6d1-0dc1-4cc5-982a-afa08b09881b admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:58.083173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7171d6d1-0dc1-4cc5-982a-afa08b09881b admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:58.085159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7171d6d1-0dc1-4cc5-982a-afa08b09881b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:58.095947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7171d6d1-0dc1-4cc5-982a-afa08b09881b admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:58.148723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'abe3610aa61448c0bc2945884b4c29c5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:58.154426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'abe3610aa61448c0bc2945884b4c29c5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:58.155331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 837/1666] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:58 2022] GET /identity/v3/roles => generated 1420 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:58.157799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7171d6d1-0dc1-4cc5-982a-afa08b09881b admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:58.167813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'abe3610aa61448c0bc2945884b4c29c5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:58.168868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'abe3610aa61448c0bc2945884b4c29c5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:58.184284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'abe3610aa61448c0bc2945884b4c29c5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:58.190356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7171d6d1-0dc1-4cc5-982a-afa08b09881b admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:58.209696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36527e01-62c9-4766-893f-1ddd7bc362c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:58.211828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 830/1667] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:29:57 2022] DELETE /identity/v3/users/abe3610aa61448c0bc2945884b4c29c5 => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:29:58.212774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36527e01-62c9-4766-893f-1ddd7bc362c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:58.214552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36527e01-62c9-4766-893f-1ddd7bc362c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:58.215056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36527e01-62c9-4766-893f-1ddd7bc362c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:58.215731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36527e01-62c9-4766-893f-1ddd7bc362c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:58.217424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36527e01-62c9-4766-893f-1ddd7bc362c1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:58.240083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 838/1668] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:58 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:58.241258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:58.242807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:58.391589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfd15bf-98b9-443d-bb02-f28d0f4bd927 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:58.399959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e684f5-7721-4cff-a2e6-119255ad11b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:58.400924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e684f5-7721-4cff-a2e6-119255ad11b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:58.401764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e684f5-7721-4cff-a2e6-119255ad11b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:58.404117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e684f5-7721-4cff-a2e6-119255ad11b8 admin admin] PATH_INFO: `/v3/projects/ff06a1beea674d5fa45de8dc86457141/users/b2c23dfbd6584999999d9725b62699dd/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:58.426396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e684f5-7721-4cff-a2e6-119255ad11b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff06a1beea674d5fa45de8dc86457141, user_id=b2c23dfbd6584999999d9725b62699dd, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:58.432055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e684f5-7721-4cff-a2e6-119255ad11b8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:58.478960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 831/1669] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:58 2022] PUT /identity/v3/projects/ff06a1beea674d5fa45de8dc86457141/users/b2c23dfbd6584999999d9725b62699dd/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:29:58.498890 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60a712b-dbb8-4a8e-af9a-67bae699f1dd None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:58.504328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60a712b-dbb8-4a8e-af9a-67bae699f1dd None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:58.505224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60a712b-dbb8-4a8e-af9a-67bae699f1dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:58.548555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c60a712b-dbb8-4a8e-af9a-67bae699f1dd None None] MFA Rules not processed for user `b2c23dfbd6584999999d9725b62699dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:58.550057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c60a712b-dbb8-4a8e-af9a-67bae699f1dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:58.640242 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e684f5-7721-4cff-a2e6-119255ad11b8 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:58.664309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 839/1670] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:58 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:29:58.748450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e684f5-7721-4cff-a2e6-119255ad11b8 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:58.755058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e1d49c-dc6d-4adc-997f-ec09ebf81bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:58.755925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e1d49c-dc6d-4adc-997f-ec09ebf81bcc admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:58.756374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e1d49c-dc6d-4adc-997f-ec09ebf81bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:58.756774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e1d49c-dc6d-4adc-997f-ec09ebf81bcc admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:58.757600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e1d49c-dc6d-4adc-997f-ec09ebf81bcc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:58.759722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e1d49c-dc6d-4adc-997f-ec09ebf81bcc admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:58.781081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:58.816029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60a712b-dbb8-4a8e-af9a-67bae699f1dd None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:58.846064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 832/1671] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:29:58 2022] POST /identity/v3/projects => generated 405 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:58.849682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60a712b-dbb8-4a8e-af9a-67bae699f1dd None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:58.910980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:58.960165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8028639e-344d-49f7-8b18-c0ec7ea0ee36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:58.960805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8028639e-344d-49f7-8b18-c0ec7ea0ee36 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:58.961078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8028639e-344d-49f7-8b18-c0ec7ea0ee36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:58.961347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8028639e-344d-49f7-8b18-c0ec7ea0ee36 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:58.962048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8028639e-344d-49f7-8b18-c0ec7ea0ee36 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:58.965000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8028639e-344d-49f7-8b18-c0ec7ea0ee36 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:59.024830 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e1d49c-dc6d-4adc-997f-ec09ebf81bcc admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:59.060847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 840/1672] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:29:58 2022] POST /identity/v3/users => generated 309 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:29:59.066336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e1d49c-dc6d-4adc-997f-ec09ebf81bcc admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:59.072864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfc4446-6fc4-47fc-bda1-5d28f4681c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:59.073709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfc4446-6fc4-47fc-bda1-5d28f4681c10 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:59.074226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfc4446-6fc4-47fc-bda1-5d28f4681c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:59.074704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfc4446-6fc4-47fc-bda1-5d28f4681c10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:59.075802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfc4446-6fc4-47fc-bda1-5d28f4681c10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:59.078046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfc4446-6fc4-47fc-bda1-5d28f4681c10 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:59.104289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8028639e-344d-49f7-8b18-c0ec7ea0ee36 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:59.105371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 833/1673] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:59 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:59.139629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8028639e-344d-49f7-8b18-c0ec7ea0ee36 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:59.146685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d130b5e-8c01-4f02-b442-3356e9016273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:59.147404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d130b5e-8c01-4f02-b442-3356e9016273 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:59.147692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d130b5e-8c01-4f02-b442-3356e9016273 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:59.147964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d130b5e-8c01-4f02-b442-3356e9016273 admin admin] PATH_INFO: `/v3/projects/2e36b9271b75418183a048f0b489517b/users/d10ba82a3c4f4383a1ce5a8cb3043034/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:59.173959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d130b5e-8c01-4f02-b442-3356e9016273 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e36b9271b75418183a048f0b489517b, user_id=d10ba82a3c4f4383a1ce5a8cb3043034, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:59.177336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d130b5e-8c01-4f02-b442-3356e9016273 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:59.207858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 841/1674] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:59 2022] PUT /identity/v3/projects/2e36b9271b75418183a048f0b489517b/users/d10ba82a3c4f4383a1ce5a8cb3043034/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:59.213005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfc4446-6fc4-47fc-bda1-5d28f4681c10 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:59.235074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfc4446-6fc4-47fc-bda1-5d28f4681c10 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:59.303039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d130b5e-8c01-4f02-b442-3356e9016273 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:59.305873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5d130b5e-8c01-4f02-b442-3356e9016273 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:59.395063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06184c1-3832-4352-aeb6-4ba41780e128 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:59.395755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06184c1-3832-4352-aeb6-4ba41780e128 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:59.396186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06184c1-3832-4352-aeb6-4ba41780e128 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:59.396636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06184c1-3832-4352-aeb6-4ba41780e128 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:59.397968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06184c1-3832-4352-aeb6-4ba41780e128 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:59.399792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06184c1-3832-4352-aeb6-4ba41780e128 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:59.415580 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 834/1675] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:29:59 2022] GET /identity/v3/roles => generated 1420 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:29:59.420768 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06184c1-3832-4352-aeb6-4ba41780e128 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:29:59.440610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06184c1-3832-4352-aeb6-4ba41780e128 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:59.460427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b81d5af-dadf-480f-b644-1616f0a900a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:59.461212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b81d5af-dadf-480f-b644-1616f0a900a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:59.461752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b81d5af-dadf-480f-b644-1616f0a900a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:59.462412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b81d5af-dadf-480f-b644-1616f0a900a4 admin admin] PATH_INFO: `/v3/projects/2e36b9271b75418183a048f0b489517b/users/d10ba82a3c4f4383a1ce5a8cb3043034/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:59.469889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b81d5af-dadf-480f-b644-1616f0a900a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e36b9271b75418183a048f0b489517b, user_id=d10ba82a3c4f4383a1ce5a8cb3043034, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:59.477278 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b81d5af-dadf-480f-b644-1616f0a900a4 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:59.509587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 835/1676] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:29:59 2022] PUT /identity/v3/projects/2e36b9271b75418183a048f0b489517b/users/d10ba82a3c4f4383a1ce5a8cb3043034/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:59.519920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688e8bf6-7047-4c3c-b493-8adb30845b0f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:59.520286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688e8bf6-7047-4c3c-b493-8adb30845b0f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:59.520998 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688e8bf6-7047-4c3c-b493-8adb30845b0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:59.554032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d130b5e-8c01-4f02-b442-3356e9016273 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:29:59.601880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-688e8bf6-7047-4c3c-b493-8adb30845b0f None None] MFA Rules not processed for user `d10ba82a3c4f4383a1ce5a8cb3043034`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:29:59.605765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-688e8bf6-7047-4c3c-b493-8adb30845b0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:29:59.785131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:29:59.788245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:59.788508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:59.791385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] PATH_INFO: `/v3/projects/a0e48f36f5434187b7b14718a5ffe290` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:29:59.798925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0e48f36f5434187b7b14718a5ffe290)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:29:59.804216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:29:59.856101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 836/1677] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:59 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:29:59.933538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0e48f36f5434187b7b14718a5ffe290', 'request_id': 'req-784565d4-23d8-415d-96b8-3c5fd10268d3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:59.942030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0e48f36f5434187b7b14718a5ffe290', 'request_id': 'req-784565d4-23d8-415d-96b8-3c5fd10268d3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:59.945040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:29:59.959135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0e48f36f5434187b7b14718a5ffe290', 'request_id': 'req-784565d4-23d8-415d-96b8-3c5fd10268d3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:59.986904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] The token cache is being invalidate because project a0e48f36f5434187b7b14718a5ffe290 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:29:59.987177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:59.987966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-784565d4-23d8-415d-96b8-3c5fd10268d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:29:59.992502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 842/1678] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:29:59 2022] DELETE /identity/v3/projects/a0e48f36f5434187b7b14718a5ffe290 => generated 0 bytes in 690 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:29:59.998887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c30adb5-74c1-4bfc-ad71-a2575e4dac60 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:29:59.999112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c30adb5-74c1-4bfc-ad71-a2575e4dac60 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:29:59.999391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c30adb5-74c1-4bfc-ad71-a2575e4dac60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:00.056753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7c30adb5-74c1-4bfc-ad71-a2575e4dac60 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:00.058399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7c30adb5-74c1-4bfc-ad71-a2575e4dac60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:00.190054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:00.246792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 837/1679] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:29:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:00.256497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c30adb5-74c1-4bfc-ad71-a2575e4dac60 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:00.256669 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda157c1-f302-432f-84a4-cee976d2fa0b None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:00.256745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda157c1-f302-432f-84a4-cee976d2fa0b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:00.257197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda157c1-f302-432f-84a4-cee976d2fa0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:00.259099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7c30adb5-74c1-4bfc-ad71-a2575e4dac60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:00.301597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-eda157c1-f302-432f-84a4-cee976d2fa0b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:00.304276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-eda157c1-f302-432f-84a4-cee976d2fa0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:00.327629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 843/1680] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:00.332930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda157c1-f302-432f-84a4-cee976d2fa0b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:00.394747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda157c1-f302-432f-84a4-cee976d2fa0b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:00.409304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa74afd8-b3a5-4d74-8b64-4c64ccdcc652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:00.411584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa74afd8-b3a5-4d74-8b64-4c64ccdcc652 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:00.412062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa74afd8-b3a5-4d74-8b64-4c64ccdcc652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:00.413286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa74afd8-b3a5-4d74-8b64-4c64ccdcc652 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:00.414891 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa74afd8-b3a5-4d74-8b64-4c64ccdcc652 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:30:00.417340 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa74afd8-b3a5-4d74-8b64-4c64ccdcc652 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:00.419730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa74afd8-b3a5-4d74-8b64-4c64ccdcc652 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:00.456626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c30adb5-74c1-4bfc-ad71-a2575e4dac60 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:00.459510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 844/1681] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:30:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:30:00.461616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa74afd8-b3a5-4d74-8b64-4c64ccdcc652 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:00.466236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb20824-e87d-4778-b544-31eb1110bc38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:00.466967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb20824-e87d-4778-b544-31eb1110bc38 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:00.467847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb20824-e87d-4778-b544-31eb1110bc38 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:00.468155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb20824-e87d-4778-b544-31eb1110bc38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:00.473059 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8fb20824-e87d-4778-b544-31eb1110bc38 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:00.488624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa74afd8-b3a5-4d74-8b64-4c64ccdcc652 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:00.502081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e97c5e-8871-4838-9156-dc622c2e2d4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:00.505094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e97c5e-8871-4838-9156-dc622c2e2d4d admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:00.505597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e97c5e-8871-4838-9156-dc622c2e2d4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:00.506046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e97c5e-8871-4838-9156-dc622c2e2d4d admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:00.507095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e97c5e-8871-4838-9156-dc622c2e2d4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:00.512847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e97c5e-8871-4838-9156-dc622c2e2d4d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:00.563525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 845/1682] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:00 2022] POST /identity/v3/projects => generated 387 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:00.568160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e97c5e-8871-4838-9156-dc622c2e2d4d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:00.732759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e97c5e-8871-4838-9156-dc622c2e2d4d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:00.747772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb20824-e87d-4778-b544-31eb1110bc38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:00.748245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862e2651-58e8-45a8-9445-643661cb271a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:00.749034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862e2651-58e8-45a8-9445-643661cb271a admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:00.749384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862e2651-58e8-45a8-9445-643661cb271a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:00.749727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862e2651-58e8-45a8-9445-643661cb271a admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:00.753576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862e2651-58e8-45a8-9445-643661cb271a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:00.756579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb20824-e87d-4778-b544-31eb1110bc38 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:00.757545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862e2651-58e8-45a8-9445-643661cb271a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:00.808000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 838/1683] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:00 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 552 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:00.814480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 846/1684] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:00 2022] POST /identity/v3/users => generated 294 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:00.817239 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb20824-e87d-4778-b544-31eb1110bc38 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:00.857434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb20824-e87d-4778-b544-31eb1110bc38 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:00.870735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0529bb6-468d-4122-a7d9-1491888a5a15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:00.871490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0529bb6-468d-4122-a7d9-1491888a5a15 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:00.871829 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0529bb6-468d-4122-a7d9-1491888a5a15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:00.872106 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0529bb6-468d-4122-a7d9-1491888a5a15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:00.872768 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0529bb6-468d-4122-a7d9-1491888a5a15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:00.876594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0529bb6-468d-4122-a7d9-1491888a5a15 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:00.946909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 839/1685] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:00 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:30:00.949455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862e2651-58e8-45a8-9445-643661cb271a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:00.973302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862e2651-58e8-45a8-9445-643661cb271a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:00.991510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db89c1a-112c-4a94-ac26-4398a39af11d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:00.992174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db89c1a-112c-4a94-ac26-4398a39af11d admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:00.992473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db89c1a-112c-4a94-ac26-4398a39af11d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:00.992731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db89c1a-112c-4a94-ac26-4398a39af11d admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:00.993389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db89c1a-112c-4a94-ac26-4398a39af11d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:00.995089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db89c1a-112c-4a94-ac26-4398a39af11d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:01.025647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 847/1686] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:00 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:01.026997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0529bb6-468d-4122-a7d9-1491888a5a15 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:01.041029 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0529bb6-468d-4122-a7d9-1491888a5a15 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:01.070941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb814b7b-74c0-4c9d-987b-2b2865121814 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:01.071653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb814b7b-74c0-4c9d-987b-2b2865121814 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:01.071963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb814b7b-74c0-4c9d-987b-2b2865121814 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:01.072240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb814b7b-74c0-4c9d-987b-2b2865121814 admin admin] PATH_INFO: `/v3/projects/fb8e5ca8b9494b1ea1820feeee64febf/users/d81ee1acc093471581bfab1a2db7fbbe/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:01.119816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb814b7b-74c0-4c9d-987b-2b2865121814 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb8e5ca8b9494b1ea1820feeee64febf, user_id=d81ee1acc093471581bfab1a2db7fbbe, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:01.122192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb814b7b-74c0-4c9d-987b-2b2865121814 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:01.162623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 840/1687] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:01 2022] PUT /identity/v3/projects/fb8e5ca8b9494b1ea1820feeee64febf/users/d81ee1acc093471581bfab1a2db7fbbe/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:01.174332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f79d01f-ee38-49be-8ee8-6e7fc4a6b6c7 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:01.174676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f79d01f-ee38-49be-8ee8-6e7fc4a6b6c7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:01.175072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f79d01f-ee38-49be-8ee8-6e7fc4a6b6c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:01.243864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-5f79d01f-ee38-49be-8ee8-6e7fc4a6b6c7 None None] MFA Rules not processed for user `d81ee1acc093471581bfab1a2db7fbbe`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:01.245137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5f79d01f-ee38-49be-8ee8-6e7fc4a6b6c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:01.382850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 848/1688] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:01 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:01.387383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb814b7b-74c0-4c9d-987b-2b2865121814 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:01.471384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8335902e-0a70-409d-9c03-8d5836cc56ef None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:01.471562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8335902e-0a70-409d-9c03-8d5836cc56ef None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:01.471817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8335902e-0a70-409d-9c03-8d5836cc56ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:01.506293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb814b7b-74c0-4c9d-987b-2b2865121814 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:01.513259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded0459-a73a-4f61-9146-9e66c53bcc90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:01.514408 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded0459-a73a-4f61-9146-9e66c53bcc90 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:01.514859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded0459-a73a-4f61-9146-9e66c53bcc90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:01.515312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded0459-a73a-4f61-9146-9e66c53bcc90 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:01.516521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fded0459-a73a-4f61-9146-9e66c53bcc90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:01.518678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fded0459-a73a-4f61-9146-9e66c53bcc90 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:01.542038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8335902e-0a70-409d-9c03-8d5836cc56ef None None] MFA Rules not processed for user `c02aec8271af4bf7aea728ff92fa555b`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:01.544598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8335902e-0a70-409d-9c03-8d5836cc56ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:01.547393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:01.555400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=1,keystone:SELECT=95,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:01.592888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 841/1689] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:01 2022] POST /identity/v3/projects => generated 385 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:01.594050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded0459-a73a-4f61-9146-9e66c53bcc90 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:01.707889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 849/1690] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:01.709485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8335902e-0a70-409d-9c03-8d5836cc56ef None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:01.711635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8335902e-0a70-409d-9c03-8d5836cc56ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:01.721457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded0459-a73a-4f61-9146-9e66c53bcc90 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:01.735451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36a4811-284e-4e4f-85e0-de5238ee6ed8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:01.736338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36a4811-284e-4e4f-85e0-de5238ee6ed8 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:01.738884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36a4811-284e-4e4f-85e0-de5238ee6ed8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:01.739257 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36a4811-284e-4e4f-85e0-de5238ee6ed8 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:01.740135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36a4811-284e-4e4f-85e0-de5238ee6ed8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:01.742332 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36a4811-284e-4e4f-85e0-de5238ee6ed8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:01.791218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 842/1691] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:01 2022] POST /identity/v3/users => generated 299 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:01.792627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36a4811-284e-4e4f-85e0-de5238ee6ed8 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:01.821127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36a4811-284e-4e4f-85e0-de5238ee6ed8 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:01.822339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8335902e-0a70-409d-9c03-8d5836cc56ef None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:01.842715 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07f7e0-e80a-47c4-8de2-1212b2667da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:01.843350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07f7e0-e80a-47c4-8de2-1212b2667da2 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:01.843626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07f7e0-e80a-47c4-8de2-1212b2667da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:01.852746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff07f7e0-e80a-47c4-8de2-1212b2667da2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:01.854565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff07f7e0-e80a-47c4-8de2-1212b2667da2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:01.863249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff07f7e0-e80a-47c4-8de2-1212b2667da2 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:01.885409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 843/1692] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:01 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:01.903858 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07f7e0-e80a-47c4-8de2-1212b2667da2 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:01.908581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9295664b-fc09-4c6b-9c5d-0cc71648310d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:01.909668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9295664b-fc09-4c6b-9c5d-0cc71648310d service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:01.910021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9295664b-fc09-4c6b-9c5d-0cc71648310d service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:01.910384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9295664b-fc09-4c6b-9c5d-0cc71648310d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:01.917171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff07f7e0-e80a-47c4-8de2-1212b2667da2 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:01.925149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc79ba1-04d2-4ad2-96f8-4796fcef06c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:01.926136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc79ba1-04d2-4ad2-96f8-4796fcef06c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:01.926403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc79ba1-04d2-4ad2-96f8-4796fcef06c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:01.926671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc79ba1-04d2-4ad2-96f8-4796fcef06c3 admin admin] PATH_INFO: `/v3/projects/1ae59edd230b4545b30ea11b4bd1ae37/users/c1c61f2a53444b6ebb4ee31ed183daa8/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:01.954086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc79ba1-04d2-4ad2-96f8-4796fcef06c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ae59edd230b4545b30ea11b4bd1ae37, user_id=c1c61f2a53444b6ebb4ee31ed183daa8, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:01.959660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:01.962428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc79ba1-04d2-4ad2-96f8-4796fcef06c3 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:01.995801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 844/1693] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:01 2022] PUT /identity/v3/projects/1ae59edd230b4545b30ea11b4bd1ae37/users/c1c61f2a53444b6ebb4ee31ed183daa8/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:02.002862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc79ba1-04d2-4ad2-96f8-4796fcef06c3 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:02.003635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9295664b-fc09-4c6b-9c5d-0cc71648310d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:02.011119 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9295664b-fc09-4c6b-9c5d-0cc71648310d service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:02.029192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 850/1694] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:01 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:02.160443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc79ba1-04d2-4ad2-96f8-4796fcef06c3 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:02.173158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ee58bb-bdfa-4d6c-902a-03b0bd06d28a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:02.173393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ee58bb-bdfa-4d6c-902a-03b0bd06d28a admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:02.173631 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ee58bb-bdfa-4d6c-902a-03b0bd06d28a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:02.173878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ee58bb-bdfa-4d6c-902a-03b0bd06d28a admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:02.174550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ee58bb-bdfa-4d6c-902a-03b0bd06d28a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:02.180051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ee58bb-bdfa-4d6c-902a-03b0bd06d28a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:02.203030 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 845/1695] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:01 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:02.203921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9295664b-fc09-4c6b-9c5d-0cc71648310d service nova] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:02.222085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9295664b-fc09-4c6b-9c5d-0cc71648310d service nova] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:02.234696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3554a1-1a93-4373-ac42-dea646bd7908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:02.235394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3554a1-1a93-4373-ac42-dea646bd7908 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:02.235834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3554a1-1a93-4373-ac42-dea646bd7908 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:02.236110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3554a1-1a93-4373-ac42-dea646bd7908 admin admin] PATH_INFO: `/v3/projects/1ae59edd230b4545b30ea11b4bd1ae37/users/c1c61f2a53444b6ebb4ee31ed183daa8/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:02.239610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3554a1-1a93-4373-ac42-dea646bd7908 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ae59edd230b4545b30ea11b4bd1ae37, user_id=c1c61f2a53444b6ebb4ee31ed183daa8, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:02.242981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3554a1-1a93-4373-ac42-dea646bd7908 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:02.262019 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 851/1696] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:02 2022] PUT /identity/v3/projects/1ae59edd230b4545b30ea11b4bd1ae37/users/c1c61f2a53444b6ebb4ee31ed183daa8/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:02.273034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb7bb3b-31bd-48ee-a56b-214235987b56 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:02.275512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb7bb3b-31bd-48ee-a56b-214235987b56 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:02.275894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb7bb3b-31bd-48ee-a56b-214235987b56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:02.320705 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8eb7bb3b-31bd-48ee-a56b-214235987b56 None None] MFA Rules not processed for user `c1c61f2a53444b6ebb4ee31ed183daa8`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:02.322139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8eb7bb3b-31bd-48ee-a56b-214235987b56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:02.383612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:02.459763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 846/1697] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:02 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:02.474819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3554a1-1a93-4373-ac42-dea646bd7908 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:02.477716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4f3554a1-1a93-4373-ac42-dea646bd7908 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:02.621172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3554a1-1a93-4373-ac42-dea646bd7908 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:02.629263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27329d8c-993c-47c6-9ca6-44d7bf0fe33b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:02.629919 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27329d8c-993c-47c6-9ca6-44d7bf0fe33b service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:02.630235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27329d8c-993c-47c6-9ca6-44d7bf0fe33b service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:02.630506 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27329d8c-993c-47c6-9ca6-44d7bf0fe33b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:02.653731 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27329d8c-993c-47c6-9ca6-44d7bf0fe33b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:02.655218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27329d8c-993c-47c6-9ca6-44d7bf0fe33b service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:02.677748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 852/1698] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:04.169964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb7bb3b-31bd-48ee-a56b-214235987b56 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:04.172771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8eb7bb3b-31bd-48ee-a56b-214235987b56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:04.188973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27329d8c-993c-47c6-9ca6-44d7bf0fe33b service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:04.354743 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb7bb3b-31bd-48ee-a56b-214235987b56 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:04.360681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:04.361347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:04.361630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:04.361905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] PATH_INFO: `/v3/users/b2c23dfbd6584999999d9725b62699dd` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:04.363270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2c23dfbd6584999999d9725b62699dd)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:04.365237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:04.379030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27329d8c-993c-47c6-9ca6-44d7bf0fe33b service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:04.392164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:04.392929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:04.393222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:04.395063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] PATH_INFO: `/v3/users/d81ee1acc093471581bfab1a2db7fbbe` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:04.398210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d81ee1acc093471581bfab1a2db7fbbe)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:04.408198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:04.432306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2c23dfbd6584999999d9725b62699dd'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:04.433298 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2c23dfbd6584999999d9725b62699dd'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:04.443391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2c23dfbd6584999999d9725b62699dd'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:04.445972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2c23dfbd6584999999d9725b62699dd'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:04.463640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-dca06576-b2fa-4955-a4a6-ece4cc886585 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2c23dfbd6584999999d9725b62699dd'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:04.497407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd81ee1acc093471581bfab1a2db7fbbe'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:04.513166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd81ee1acc093471581bfab1a2db7fbbe'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:04.514180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd81ee1acc093471581bfab1a2db7fbbe'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:04.514980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd81ee1acc093471581bfab1a2db7fbbe'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:04.531459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 847/1699] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:04 2022] DELETE /identity/v3/users/b2c23dfbd6584999999d9725b62699dd => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:04.533259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fa6f76d3-e45f-4456-bdff-2f65ba1f33c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd81ee1acc093471581bfab1a2db7fbbe'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:04.536122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef9f357-c416-43e5-8889-d3e7a06fabd3 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:04.536583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef9f357-c416-43e5-8889-d3e7a06fabd3 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:04.536999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef9f357-c416-43e5-8889-d3e7a06fabd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:04.546518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:04.573428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 853/1700] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:04 2022] DELETE /identity/v3/users/d81ee1acc093471581bfab1a2db7fbbe => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:30:04.578961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc40ac5d-5e6b-4161-8974-41c5e4e13005 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:04.579826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc40ac5d-5e6b-4161-8974-41c5e4e13005 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:04.580831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc40ac5d-5e6b-4161-8974-41c5e4e13005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:04.594864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-bef9f357-c416-43e5-8889-d3e7a06fabd3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:04.596375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-bef9f357-c416-43e5-8889-d3e7a06fabd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:04.607732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-fc40ac5d-5e6b-4161-8974-41c5e4e13005 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:04.609341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-fc40ac5d-5e6b-4161-8974-41c5e4e13005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:04.693740 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 854/1701] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:04.710978 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:04.715809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc40ac5d-5e6b-4161-8974-41c5e4e13005 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:04.722547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-fc40ac5d-5e6b-4161-8974-41c5e4e13005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:04.753337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 848/1702] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:04.774518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef9f357-c416-43e5-8889-d3e7a06fabd3 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:04.776753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-bef9f357-c416-43e5-8889-d3e7a06fabd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:04.905329 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc40ac5d-5e6b-4161-8974-41c5e4e13005 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:04.917972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2a83a5-fc9d-49ef-ab36-9689c25efe77 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:04.918658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2a83a5-fc9d-49ef-ab36-9689c25efe77 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:04.918938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2a83a5-fc9d-49ef-ab36-9689c25efe77 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:04.923800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2a83a5-fc9d-49ef-ab36-9689c25efe77 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:04.928280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef9f357-c416-43e5-8889-d3e7a06fabd3 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:04.939518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18082fa1-41d0-4f15-a357-a25080953635 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:04.940427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18082fa1-41d0-4f15-a357-a25080953635 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:04.941155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18082fa1-41d0-4f15-a357-a25080953635 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:04.941803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18082fa1-41d0-4f15-a357-a25080953635 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:04.958451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2a83a5-fc9d-49ef-ab36-9689c25efe77 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:04.960121 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2a83a5-fc9d-49ef-ab36-9689c25efe77 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:04.967271 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18082fa1-41d0-4f15-a357-a25080953635 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:04.968996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18082fa1-41d0-4f15-a357-a25080953635 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:04.986807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 855/1703] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:05.003682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 849/1704] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:30:05.676843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dad45e-afd3-425c-a347-93b8c2deb981 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:05.677396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dad45e-afd3-425c-a347-93b8c2deb981 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:05.677478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dad45e-afd3-425c-a347-93b8c2deb981 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:05.733218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-15dad45e-afd3-425c-a347-93b8c2deb981 None None] MFA Rules not processed for user `c02aec8271af4bf7aea728ff92fa555b`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:05.738577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-15dad45e-afd3-425c-a347-93b8c2deb981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:05.916669 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 856/1705] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:05 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:06.657339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18082fa1-41d0-4f15-a357-a25080953635 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:06.660442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-18082fa1-41d0-4f15-a357-a25080953635 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:06.690991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18082fa1-41d0-4f15-a357-a25080953635 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:06.700924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:06.701898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:06.702458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:06.726927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] PATH_INFO: `/v3/projects/ff06a1beea674d5fa45de8dc86457141` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:06.732098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff06a1beea674d5fa45de8dc86457141)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:06.738249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:06.862069 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff06a1beea674d5fa45de8dc86457141', 'request_id': 'req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:06.879132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff06a1beea674d5fa45de8dc86457141', 'request_id': 'req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:06.903791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff06a1beea674d5fa45de8dc86457141', 'request_id': 'req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:06.918917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] The token cache is being invalidate because project ff06a1beea674d5fa45de8dc86457141 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:06.919919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:06.920305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:06.922649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 850/1706] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:06 2022] DELETE /identity/v3/projects/ff06a1beea674d5fa45de8dc86457141 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:06.926572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dad45e-afd3-425c-a347-93b8c2deb981 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:06.943570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dad45e-afd3-425c-a347-93b8c2deb981 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:07.060226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:07.113589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:07.117762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:07.118057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:07.118489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] PATH_INFO: `/v3/users/d10ba82a3c4f4383a1ce5a8cb3043034` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:07.126715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d10ba82a3c4f4383a1ce5a8cb3043034)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:07.129801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:07.269405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd10ba82a3c4f4383a1ce5a8cb3043034'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:07.286842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:07.287580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd10ba82a3c4f4383a1ce5a8cb3043034'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:07.288275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:07.288615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd10ba82a3c4f4383a1ce5a8cb3043034'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:07.289601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd10ba82a3c4f4383a1ce5a8cb3043034'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:07.318291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-ba8276da-c8cb-4a1c-8274-3d845c0db53d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd10ba82a3c4f4383a1ce5a8cb3043034'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:07.327283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f968ebc-5518-4fc8-b5f2-576a1eb39c7a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:07.355427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:07.356836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:07.359131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:07.360415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] PATH_INFO: `/v3/projects/fb8e5ca8b9494b1ea1820feeee64febf` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:07.371787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb8e5ca8b9494b1ea1820feeee64febf)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:07.372209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 857/1707] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:06 2022] DELETE /identity/v3/users/d10ba82a3c4f4383a1ce5a8cb3043034 => generated 0 bytes in 446 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:07.376986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:07.597128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb8e5ca8b9494b1ea1820feeee64febf', 'request_id': 'req-4814eb2b-ca32-4844-9efb-4dcc5134362f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:07.647026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11cbf4-48dc-48d6-84dc-45771a9800de None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:07.647520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11cbf4-48dc-48d6-84dc-45771a9800de None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:07.647681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb8e5ca8b9494b1ea1820feeee64febf', 'request_id': 'req-4814eb2b-ca32-4844-9efb-4dcc5134362f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:07.648148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11cbf4-48dc-48d6-84dc-45771a9800de None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:07.650292 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 858/1708] 10.222.0.16 () {54 vars in 912 bytes} [Tue Sep 13 15:30:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:30:07.678977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f727430-3f4d-4f9c-b688-90504e24ea58 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:07.679323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb8e5ca8b9494b1ea1820feeee64febf', 'request_id': 'req-4814eb2b-ca32-4844-9efb-4dcc5134362f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:07.681002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f727430-3f4d-4f9c-b688-90504e24ea58 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:07.684806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f727430-3f4d-4f9c-b688-90504e24ea58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:07.688563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:07.726969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] The token cache is being invalidate because project fb8e5ca8b9494b1ea1820feeee64febf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:07.727430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:07.732221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:07.741776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 851/1709] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:07 2022] DELETE /identity/v3/projects/fb8e5ca8b9494b1ea1820feeee64febf => generated 0 bytes in 452 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:07.743104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:07.745459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:07.810849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-1f727430-3f4d-4f9c-b688-90504e24ea58 None None] MFA Rules not processed for user `171e916244804add8f18b44b77b03d72`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:07.820871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1f727430-3f4d-4f9c-b688-90504e24ea58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:07.954080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814eb2b-ca32-4844-9efb-4dcc5134362f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:07.966291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:07.966957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:07.967234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:07.968152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] PATH_INFO: `/v3/users/c1c61f2a53444b6ebb4ee31ed183daa8` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:07.972386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1c61f2a53444b6ebb4ee31ed183daa8)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:07.975875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 859/1710] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:30:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:30:07.977470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2f0083-616c-4704-97b0-81a0fd7b016f None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:07.977762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2f0083-616c-4704-97b0-81a0fd7b016f None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:07.978031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2f0083-616c-4704-97b0-81a0fd7b016f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:07.979169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:08.048081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4a2f0083-616c-4704-97b0-81a0fd7b016f None None] MFA Rules not processed for user `c8dcc087badb43b7a18d294a6e316fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:08.049728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4a2f0083-616c-4704-97b0-81a0fd7b016f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:08.086325 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1c61f2a53444b6ebb4ee31ed183daa8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:08.087221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1c61f2a53444b6ebb4ee31ed183daa8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:08.109716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1c61f2a53444b6ebb4ee31ed183daa8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:08.112532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1c61f2a53444b6ebb4ee31ed183daa8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:08.138285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1c61f2a53444b6ebb4ee31ed183daa8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:08.198916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 852/1711] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:07 2022] DELETE /identity/v3/users/c1c61f2a53444b6ebb4ee31ed183daa8 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:08.198916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:08.201337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:08.204760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:08.298230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 860/1712] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:08.307921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff87cca4-2abe-48f9-b508-76dbb8bdaca2 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:08.309039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff87cca4-2abe-48f9-b508-76dbb8bdaca2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:08.309537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff87cca4-2abe-48f9-b508-76dbb8bdaca2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:08.379195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-ff87cca4-2abe-48f9-b508-76dbb8bdaca2 None None] MFA Rules not processed for user `c8dcc087badb43b7a18d294a6e316fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:08.380517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ff87cca4-2abe-48f9-b508-76dbb8bdaca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:08.407182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2712a5c-6225-4a78-92c0-7490a23a248a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:08.417909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edeb16f-9914-40cc-af23-25d84e404d72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:08.419233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edeb16f-9914-40cc-af23-25d84e404d72 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:08.419233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edeb16f-9914-40cc-af23-25d84e404d72 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:08.419691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edeb16f-9914-40cc-af23-25d84e404d72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:08.422899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-5edeb16f-9914-40cc-af23-25d84e404d72 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:08.466489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edeb16f-9914-40cc-af23-25d84e404d72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:08.471610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edeb16f-9914-40cc-af23-25d84e404d72 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:08.520794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 853/1713] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:08.523407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edeb16f-9914-40cc-af23-25d84e404d72 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:08.527033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-5edeb16f-9914-40cc-af23-25d84e404d72 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:08.536858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 861/1714] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:08 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:08.548999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff87cca4-2abe-48f9-b508-76dbb8bdaca2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:08.552445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ff87cca4-2abe-48f9-b508-76dbb8bdaca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:08.605603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:08.751885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edeb16f-9914-40cc-af23-25d84e404d72 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:08.761565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:08.762565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:08.762912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:08.763901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] PATH_INFO: `/v3/projects/2e36b9271b75418183a048f0b489517b` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:08.765775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e36b9271b75418183a048f0b489517b)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:08.767936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:08.865168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff87cca4-2abe-48f9-b508-76dbb8bdaca2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:08.880411 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e36b9271b75418183a048f0b489517b', 'request_id': 'req-aec75dcd-4df5-4127-a585-0d232fffe028'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:08.913494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e36b9271b75418183a048f0b489517b', 'request_id': 'req-aec75dcd-4df5-4127-a585-0d232fffe028'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:08.945094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e36b9271b75418183a048f0b489517b', 'request_id': 'req-aec75dcd-4df5-4127-a585-0d232fffe028'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:08.973860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] The token cache is being invalidate because project 2e36b9271b75418183a048f0b489517b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:08.982055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:08.984383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-aec75dcd-4df5-4127-a585-0d232fffe028 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:08.992951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 854/1715] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:08 2022] DELETE /identity/v3/projects/2e36b9271b75418183a048f0b489517b => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:09.001329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1342d0-cbcd-4d07-a5d8-3d816db6752d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:09.003423 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1342d0-cbcd-4d07-a5d8-3d816db6752d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:09.004409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1342d0-cbcd-4d07-a5d8-3d816db6752d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:09.044401 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f97827-5bd1-457f-b7c2-c677cc35b7a5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:09.045051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f97827-5bd1-457f-b7c2-c677cc35b7a5 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:09.051284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f97827-5bd1-457f-b7c2-c677cc35b7a5 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:09.054348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f97827-5bd1-457f-b7c2-c677cc35b7a5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:09.061070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-95f97827-5bd1-457f-b7c2-c677cc35b7a5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:09.078920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7a1342d0-cbcd-4d07-a5d8-3d816db6752d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:09.080173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7a1342d0-cbcd-4d07-a5d8-3d816db6752d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:09.193733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:09.383817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f97827-5bd1-457f-b7c2-c677cc35b7a5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:09.385477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f97827-5bd1-457f-b7c2-c677cc35b7a5 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:09.390792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 855/1716] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 394 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:09.398760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8afd5bb-9e69-4891-91f5-852d25c66376 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:09.398760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8afd5bb-9e69-4891-91f5-852d25c66376 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:09.399055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8afd5bb-9e69-4891-91f5-852d25c66376 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:09.414307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 862/1717] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 869 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:30:09.478244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c8afd5bb-9e69-4891-91f5-852d25c66376 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:09.483981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c8afd5bb-9e69-4891-91f5-852d25c66376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:09.534168 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 856/1718] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:09.537425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f97827-5bd1-457f-b7c2-c677cc35b7a5 service nova] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:09.539294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-95f97827-5bd1-457f-b7c2-c677cc35b7a5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:09.549174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8afd5bb-9e69-4891-91f5-852d25c66376 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:09.564392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f97827-5bd1-457f-b7c2-c677cc35b7a5 service nova] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:09.571141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8afd5bb-9e69-4891-91f5-852d25c66376 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:09.575395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:09.578314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:09.578915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:09.580169 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] PATH_INFO: `/v3/projects/1ae59edd230b4545b30ea11b4bd1ae37` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:09.585971 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ae59edd230b4545b30ea11b4bd1ae37)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:09.600638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b6aafc-3fc8-4b49-a3d1-a833e850adcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:09.615165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b6aafc-3fc8-4b49-a3d1-a833e850adcf admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:09.616170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:09.617133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b6aafc-3fc8-4b49-a3d1-a833e850adcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:09.618087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b6aafc-3fc8-4b49-a3d1-a833e850adcf admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:09.621291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b6aafc-3fc8-4b49-a3d1-a833e850adcf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:30:09.625471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b6aafc-3fc8-4b49-a3d1-a833e850adcf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:09.628614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b6aafc-3fc8-4b49-a3d1-a833e850adcf admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:09.690776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 857/1719] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:30:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:30:09.695426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b6aafc-3fc8-4b49-a3d1-a833e850adcf admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:09.751497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b6aafc-3fc8-4b49-a3d1-a833e850adcf admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:09.784297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeff2b7-262d-4724-962a-da7d94c6b557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:09.784297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeff2b7-262d-4724-962a-da7d94c6b557 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:09.784988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeff2b7-262d-4724-962a-da7d94c6b557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:09.784988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeff2b7-262d-4724-962a-da7d94c6b557 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:09.785435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeff2b7-262d-4724-962a-da7d94c6b557 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:09.806858 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeff2b7-262d-4724-962a-da7d94c6b557 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:09.825388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ae59edd230b4545b30ea11b4bd1ae37', 'request_id': 'req-77d19ddc-1efa-428a-868f-c1332d0b1fb7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:09.835559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ae59edd230b4545b30ea11b4bd1ae37', 'request_id': 'req-77d19ddc-1efa-428a-868f-c1332d0b1fb7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:09.858557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ae59edd230b4545b30ea11b4bd1ae37', 'request_id': 'req-77d19ddc-1efa-428a-868f-c1332d0b1fb7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:09.875271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] The token cache is being invalidate because project 1ae59edd230b4545b30ea11b4bd1ae37 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:09.876947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:09.878559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:09.881389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 863/1720] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:09 2022] DELETE /identity/v3/projects/1ae59edd230b4545b30ea11b4bd1ae37 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:09.883115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:09.885389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:09.912819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 858/1721] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:09 2022] POST /identity/v3/projects => generated 365 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:09.922179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7e4569-8840-4698-95ae-822b61fdb400 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:09.922639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7e4569-8840-4698-95ae-822b61fdb400 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:09.924373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7e4569-8840-4698-95ae-822b61fdb400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:09.961394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:09.980182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-6e7e4569-8840-4698-95ae-822b61fdb400 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:09.984253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6e7e4569-8840-4698-95ae-822b61fdb400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:10.089083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d19ddc-1efa-428a-868f-c1332d0b1fb7 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:10.211801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 859/1722] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:10.211801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7e4569-8840-4698-95ae-822b61fdb400 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:10.222944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6e7e4569-8840-4698-95ae-822b61fdb400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:10.253039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7e4569-8840-4698-95ae-822b61fdb400 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:10.261634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:10.287926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8fea29-6bf7-46b9-98d7-29354ad391cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:10.288615 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8fea29-6bf7-46b9-98d7-29354ad391cb admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:10.288892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8fea29-6bf7-46b9-98d7-29354ad391cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:10.290762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8fea29-6bf7-46b9-98d7-29354ad391cb admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:10.291425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8fea29-6bf7-46b9-98d7-29354ad391cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:10.296494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8fea29-6bf7-46b9-98d7-29354ad391cb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:10.309046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a914e0-152d-483b-b5c7-2a0e8fdc0410 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:10.310394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a914e0-152d-483b-b5c7-2a0e8fdc0410 service placement] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:10.310735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a914e0-152d-483b-b5c7-2a0e8fdc0410 service placement] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:10.311072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a914e0-152d-483b-b5c7-2a0e8fdc0410 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:10.320203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-97a914e0-152d-483b-b5c7-2a0e8fdc0410 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:10.356335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a914e0-152d-483b-b5c7-2a0e8fdc0410 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:10.359911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a914e0-152d-483b-b5c7-2a0e8fdc0410 service placement] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:10.371958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 860/1723] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:10 2022] POST /identity/v3/users => generated 283 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:10.375669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71af97cc-dff8-4516-af03-10d105dee7f5 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:10.376115 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71af97cc-dff8-4516-af03-10d105dee7f5 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:10.376827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71af97cc-dff8-4516-af03-10d105dee7f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:10.444802 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 864/1724] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 563 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:10.445690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a914e0-152d-483b-b5c7-2a0e8fdc0410 service placement] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:10.469010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a914e0-152d-483b-b5c7-2a0e8fdc0410 service placement] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:10.470757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-71af97cc-dff8-4516-af03-10d105dee7f5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:10.480005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-71af97cc-dff8-4516-af03-10d105dee7f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:10.484687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286cd976-633f-4d9d-98f5-87d29b3c48f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:10.485328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286cd976-633f-4d9d-98f5-87d29b3c48f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:10.485606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286cd976-633f-4d9d-98f5-87d29b3c48f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:10.485875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286cd976-633f-4d9d-98f5-87d29b3c48f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:10.486554 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286cd976-633f-4d9d-98f5-87d29b3c48f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:10.488169 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286cd976-633f-4d9d-98f5-87d29b3c48f9 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:10.507041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 861/1725] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:10.508767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8908ac4-611f-46f1-8b22-e4675f99f02f None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:10.509162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8908ac4-611f-46f1-8b22-e4675f99f02f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:10.511890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8908ac4-611f-46f1-8b22-e4675f99f02f None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:10.513760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 862/1726] 10.222.0.16 () {54 vars in 912 bytes} [Tue Sep 13 15:30:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:30:10.514777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8908ac4-611f-46f1-8b22-e4675f99f02f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:10.521499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 865/1727] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:10 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:10.522797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea104a2f-c6c5-4efb-bdc4-d03c5b420784 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:10.523083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea104a2f-c6c5-4efb-bdc4-d03c5b420784 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:10.523364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea104a2f-c6c5-4efb-bdc4-d03c5b420784 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:10.572310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8908ac4-611f-46f1-8b22-e4675f99f02f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:10.579206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d387a945-48b1-457c-91cd-e13d2ba5d294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:10.579623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d387a945-48b1-457c-91cd-e13d2ba5d294 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:10.579923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d387a945-48b1-457c-91cd-e13d2ba5d294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:10.580250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d387a945-48b1-457c-91cd-e13d2ba5d294 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:10.581123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d387a945-48b1-457c-91cd-e13d2ba5d294 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:30:10.581619 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d387a945-48b1-457c-91cd-e13d2ba5d294 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:10.583084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-ea104a2f-c6c5-4efb-bdc4-d03c5b420784 None None] MFA Rules not processed for user `171e916244804add8f18b44b77b03d72`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:10.584780 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ea104a2f-c6c5-4efb-bdc4-d03c5b420784 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:10.586275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d387a945-48b1-457c-91cd-e13d2ba5d294 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:10.669390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 863/1728] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:30:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:30:10.671852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d387a945-48b1-457c-91cd-e13d2ba5d294 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:10.713896 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d387a945-48b1-457c-91cd-e13d2ba5d294 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:10.745567 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f3c3b8-a79a-4198-8ab4-98ef6fec7ea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:10.746273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f3c3b8-a79a-4198-8ab4-98ef6fec7ea5 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:10.746552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f3c3b8-a79a-4198-8ab4-98ef6fec7ea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:10.746822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f3c3b8-a79a-4198-8ab4-98ef6fec7ea5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:10.747471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f3c3b8-a79a-4198-8ab4-98ef6fec7ea5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:10.757199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f3c3b8-a79a-4198-8ab4-98ef6fec7ea5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:10.817277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 864/1729] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:10 2022] GET /identity/v3/roles => generated 1420 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:10.817277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f3c3b8-a79a-4198-8ab4-98ef6fec7ea5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:10.837743 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 866/1730] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:30:10 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:10.838859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea104a2f-c6c5-4efb-bdc4-d03c5b420784 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:10.853245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f3c3b8-a79a-4198-8ab4-98ef6fec7ea5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:10.878010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fd7d9e-d401-4aaf-8b1e-958c61569b1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:10.878880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fd7d9e-d401-4aaf-8b1e-958c61569b1e admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:10.879322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fd7d9e-d401-4aaf-8b1e-958c61569b1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:10.879594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fd7d9e-d401-4aaf-8b1e-958c61569b1e admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:10.884455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea104a2f-c6c5-4efb-bdc4-d03c5b420784 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:10.887782 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fd7d9e-d401-4aaf-8b1e-958c61569b1e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:10.889787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fd7d9e-d401-4aaf-8b1e-958c61569b1e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:10.898093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10599644-a709-46b7-8e15-400d7e2a18d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:10.898822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10599644-a709-46b7-8e15-400d7e2a18d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:10.899141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10599644-a709-46b7-8e15-400d7e2a18d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:10.899422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10599644-a709-46b7-8e15-400d7e2a18d7 admin admin] PATH_INFO: `/v3/projects/bb2a6edb2dc34dd29701c9d74541ed08/users/c90ae5e8c0dd48e38f7caf75f214afb4/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:10.951611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 865/1731] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:10 2022] POST /identity/v3/projects => generated 371 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:10.952730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fd7d9e-d401-4aaf-8b1e-958c61569b1e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:10.954874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10599644-a709-46b7-8e15-400d7e2a18d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb2a6edb2dc34dd29701c9d74541ed08, user_id=c90ae5e8c0dd48e38f7caf75f214afb4, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:10.961601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d1fd7d9e-d401-4aaf-8b1e-958c61569b1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:10.970269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10599644-a709-46b7-8e15-400d7e2a18d7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:11.006064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 867/1732] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:10 2022] PUT /identity/v3/projects/bb2a6edb2dc34dd29701c9d74541ed08/users/c90ae5e8c0dd48e38f7caf75f214afb4/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:11.018200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10599644-a709-46b7-8e15-400d7e2a18d7 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:11.111739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fd7d9e-d401-4aaf-8b1e-958c61569b1e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:11.325486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77e62cc-42b4-471e-900d-b246b69f06d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:11.326415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77e62cc-42b4-471e-900d-b246b69f06d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:11.327508 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77e62cc-42b4-471e-900d-b246b69f06d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:11.327716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10599644-a709-46b7-8e15-400d7e2a18d7 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:11.328554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77e62cc-42b4-471e-900d-b246b69f06d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:11.337244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:11.348419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77e62cc-42b4-471e-900d-b246b69f06d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:11.350251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77e62cc-42b4-471e-900d-b246b69f06d6 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:11.361903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 866/1733] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:11.364062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed0b96d-f149-49da-ad98-0c962d5e7b5a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:11.364502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed0b96d-f149-49da-ad98-0c962d5e7b5a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:11.364914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed0b96d-f149-49da-ad98-0c962d5e7b5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:11.372316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604c049d-7916-4d4f-8717-7a64cacb7f9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:11.372875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604c049d-7916-4d4f-8717-7a64cacb7f9b admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:11.373133 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604c049d-7916-4d4f-8717-7a64cacb7f9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:11.373401 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604c049d-7916-4d4f-8717-7a64cacb7f9b admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:11.374048 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604c049d-7916-4d4f-8717-7a64cacb7f9b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:11.378741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604c049d-7916-4d4f-8717-7a64cacb7f9b admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:11.456596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 868/1734] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:11 2022] POST /identity/v3/users => generated 286 bytes in 447 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:11.460960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193a6061-4c16-4ea5-a96e-d2e7e86e631e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:11.461421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193a6061-4c16-4ea5-a96e-d2e7e86e631e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:11.462237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193a6061-4c16-4ea5-a96e-d2e7e86e631e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:11.502593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-5ed0b96d-f149-49da-ad98-0c962d5e7b5a None None] MFA Rules not processed for user `97189d5c87c94be2bef7cff79c471c7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:11.504050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-5ed0b96d-f149-49da-ad98-0c962d5e7b5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:11.600305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-193a6061-4c16-4ea5-a96e-d2e7e86e631e None None] MFA Rules not processed for user `c90ae5e8c0dd48e38f7caf75f214afb4`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:11.604453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-193a6061-4c16-4ea5-a96e-d2e7e86e631e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:11.780376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:11.790400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 867/1735] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:11 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 427 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:11.793289 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f8b32b-ad0f-4f17-b205-bd198d149a56 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:11.793578 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f8b32b-ad0f-4f17-b205-bd198d149a56 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:11.793854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f8b32b-ad0f-4f17-b205-bd198d149a56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:11.893401 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 869/1736] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:11 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 439 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:11.898558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193a6061-4c16-4ea5-a96e-d2e7e86e631e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:11.899324 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d7f8b32b-ad0f-4f17-b205-bd198d149a56 None None] MFA Rules not processed for user `515e0f8a3a234deba5f909271b303abb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:11.901475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d7f8b32b-ad0f-4f17-b205-bd198d149a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:11.967845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193a6061-4c16-4ea5-a96e-d2e7e86e631e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:11.986217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac2f17-8913-46bb-9ff2-086a95c82645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:11.987107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac2f17-8913-46bb-9ff2-086a95c82645 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:11.989862 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac2f17-8913-46bb-9ff2-086a95c82645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:11.990355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac2f17-8913-46bb-9ff2-086a95c82645 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:11.991290 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ac2f17-8913-46bb-9ff2-086a95c82645 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:11.998440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ac2f17-8913-46bb-9ff2-086a95c82645 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:12.030243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 868/1737] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:11 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:12.030243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67103153-3a9d-4c05-b355-cfdb314b8422 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:12.030243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67103153-3a9d-4c05-b355-cfdb314b8422 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:12.030243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67103153-3a9d-4c05-b355-cfdb314b8422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:12.036195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 870/1738] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:11 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:12.047177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac2f17-8913-46bb-9ff2-086a95c82645 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:12.057501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-67103153-3a9d-4c05-b355-cfdb314b8422 None None] MFA Rules not processed for user `97189d5c87c94be2bef7cff79c471c7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:12.059755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-67103153-3a9d-4c05-b355-cfdb314b8422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:12.073228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 869/1739] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:12 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:12.074668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67103153-3a9d-4c05-b355-cfdb314b8422 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:12.075979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-67103153-3a9d-4c05-b355-cfdb314b8422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:12.078270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac2f17-8913-46bb-9ff2-086a95c82645 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:12.093806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84fcd41-15dc-4f68-8217-387ea085db05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:12.095924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84fcd41-15dc-4f68-8217-387ea085db05 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:12.096362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84fcd41-15dc-4f68-8217-387ea085db05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:12.098958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84fcd41-15dc-4f68-8217-387ea085db05 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:12.100815 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84fcd41-15dc-4f68-8217-387ea085db05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:12.103033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84fcd41-15dc-4f68-8217-387ea085db05 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:12.208133 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 871/1740] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:12 2022] POST /identity/v3/projects => generated 365 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:12.208926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84fcd41-15dc-4f68-8217-387ea085db05 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:12.232926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67103153-3a9d-4c05-b355-cfdb314b8422 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:12.239916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:12.361269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84fcd41-15dc-4f68-8217-387ea085db05 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:12.394733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2482b9-512f-4ff1-a90f-9ff8f3d7fe50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:12.395373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2482b9-512f-4ff1-a90f-9ff8f3d7fe50 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:12.395650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2482b9-512f-4ff1-a90f-9ff8f3d7fe50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:12.396063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2482b9-512f-4ff1-a90f-9ff8f3d7fe50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:12.397660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2482b9-512f-4ff1-a90f-9ff8f3d7fe50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:12.402866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2482b9-512f-4ff1-a90f-9ff8f3d7fe50 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:12.424821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacbf419-4649-4e9b-8683-e26f96abd8e0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:12.425467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacbf419-4649-4e9b-8683-e26f96abd8e0 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:12.425738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacbf419-4649-4e9b-8683-e26f96abd8e0 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:12.426004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacbf419-4649-4e9b-8683-e26f96abd8e0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:12.426454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 872/1741] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:12 2022] GET /identity/v3/roles => generated 1420 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:12.428616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2482b9-512f-4ff1-a90f-9ff8f3d7fe50 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:12.448277 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2482b9-512f-4ff1-a90f-9ff8f3d7fe50 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:12.464211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a63e7b-42b8-4a35-b61c-a910c52f8ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:12.464783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a63e7b-42b8-4a35-b61c-a910c52f8ac9 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:12.465087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a63e7b-42b8-4a35-b61c-a910c52f8ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:12.465359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a63e7b-42b8-4a35-b61c-a910c52f8ac9 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:12.465980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a63e7b-42b8-4a35-b61c-a910c52f8ac9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:12.472981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a63e7b-42b8-4a35-b61c-a910c52f8ac9 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:12.548742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 873/1742] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:12 2022] POST /identity/v3/users => generated 289 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:12.559475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a63e7b-42b8-4a35-b61c-a910c52f8ac9 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:12.614632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a63e7b-42b8-4a35-b61c-a910c52f8ac9 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:12.628990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19cbd85-a843-46cd-a82f-5fbe8cb65aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:12.629691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19cbd85-a843-46cd-a82f-5fbe8cb65aa4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:12.629966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19cbd85-a843-46cd-a82f-5fbe8cb65aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:12.630264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19cbd85-a843-46cd-a82f-5fbe8cb65aa4 admin admin] PATH_INFO: `/v3/projects/4e70bc4c4f6c429daf16241a82624849/users/c9a79828d83e4535bb46631b52b7d7ec/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:12.637068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacbf419-4649-4e9b-8683-e26f96abd8e0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:12.638762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacbf419-4649-4e9b-8683-e26f96abd8e0 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:12.709044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 870/1743] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:12 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 635 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:12.710392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacbf419-4649-4e9b-8683-e26f96abd8e0 service nova] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:12.719419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19cbd85-a843-46cd-a82f-5fbe8cb65aa4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e70bc4c4f6c429daf16241a82624849, user_id=c9a79828d83e4535bb46631b52b7d7ec, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:12.732760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19cbd85-a843-46cd-a82f-5fbe8cb65aa4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:12.767306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacbf419-4649-4e9b-8683-e26f96abd8e0 service nova] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:12.827420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 874/1744] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:12 2022] PUT /identity/v3/projects/4e70bc4c4f6c429daf16241a82624849/users/c9a79828d83e4535bb46631b52b7d7ec/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:12.829310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd71519d-e60f-4d10-bac5-a769453fba47 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:12.829553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd71519d-e60f-4d10-bac5-a769453fba47 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:12.830439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd71519d-e60f-4d10-bac5-a769453fba47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:12.964460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-dd71519d-e60f-4d10-bac5-a769453fba47 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:12.991003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-dd71519d-e60f-4d10-bac5-a769453fba47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:13.139507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7e1aaa-ecdb-4799-b8bb-0382f5d926f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:13.140175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7e1aaa-ecdb-4799-b8bb-0382f5d926f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:13.140449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7e1aaa-ecdb-4799-b8bb-0382f5d926f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:13.140999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7e1aaa-ecdb-4799-b8bb-0382f5d926f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:13.141701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7e1aaa-ecdb-4799-b8bb-0382f5d926f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:13.143736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7e1aaa-ecdb-4799-b8bb-0382f5d926f5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:13.173541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 871/1745] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:12 2022] GET /identity/v3/roles => generated 1420 bytes in 461 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:13.175365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7e1aaa-ecdb-4799-b8bb-0382f5d926f5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:13.178721 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:13.212143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 875/1746] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:13.213384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd71519d-e60f-4d10-bac5-a769453fba47 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:13.259039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd71519d-e60f-4d10-bac5-a769453fba47 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:13.278389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dde63c0-2a39-4ffc-9b87-33ce3a364f0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:13.279529 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dde63c0-2a39-4ffc-9b87-33ce3a364f0e admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:13.281407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dde63c0-2a39-4ffc-9b87-33ce3a364f0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:13.281814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dde63c0-2a39-4ffc-9b87-33ce3a364f0e admin admin] PATH_INFO: `/v3/projects/d51a59a863ab4ba1bd03f51f0a90846e/users/0a8e9d8126734ccbb8913f4976ec76ce/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:13.305310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:13.327035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dde63c0-2a39-4ffc-9b87-33ce3a364f0e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d51a59a863ab4ba1bd03f51f0a90846e, user_id=0a8e9d8126734ccbb8913f4976ec76ce, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:13.335860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dde63c0-2a39-4ffc-9b87-33ce3a364f0e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:13.421239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 876/1747] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:13 2022] PUT /identity/v3/projects/d51a59a863ab4ba1bd03f51f0a90846e/users/0a8e9d8126734ccbb8913f4976ec76ce/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:13.427364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dde63c0-2a39-4ffc-9b87-33ce3a364f0e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:13.439438 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1dde63c0-2a39-4ffc-9b87-33ce3a364f0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:13.471687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7e1aaa-ecdb-4799-b8bb-0382f5d926f5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:13.648582 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bafa5ca-67a0-4f00-b3b0-2180f1704081 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:13.649437 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafa5ca-67a0-4f00-b3b0-2180f1704081 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:13.650224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafa5ca-67a0-4f00-b3b0-2180f1704081 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:13.650224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafa5ca-67a0-4f00-b3b0-2180f1704081 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:13.653576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dde63c0-2a39-4ffc-9b87-33ce3a364f0e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:13.660274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c074e6-a758-451f-ac25-7e0ab3e46971 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:13.661184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c074e6-a758-451f-ac25-7e0ab3e46971 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:13.661745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c074e6-a758-451f-ac25-7e0ab3e46971 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:13.662267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c074e6-a758-451f-ac25-7e0ab3e46971 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:13.848380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c074e6-a758-451f-ac25-7e0ab3e46971 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:13.849185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bafa5ca-67a0-4f00-b3b0-2180f1704081 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:13.850551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c074e6-a758-451f-ac25-7e0ab3e46971 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:13.853176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bafa5ca-67a0-4f00-b3b0-2180f1704081 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:13.890549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 872/1748] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:13 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 716 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:13.894631 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bafa5ca-67a0-4f00-b3b0-2180f1704081 service nova] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:13.902536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 877/1749] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:13.921988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bafa5ca-67a0-4f00-b3b0-2180f1704081 service nova] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:13.947465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cf43d8-139a-4ac0-9c5e-398480fd5ad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:13.950379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf43d8-139a-4ac0-9c5e-398480fd5ad8 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:13.980313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf43d8-139a-4ac0-9c5e-398480fd5ad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:13.990718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf43d8-139a-4ac0-9c5e-398480fd5ad8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:13.993803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cf43d8-139a-4ac0-9c5e-398480fd5ad8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:13.999029 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cf43d8-139a-4ac0-9c5e-398480fd5ad8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:14.032543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 873/1750] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:13 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:14.037305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c074e6-a758-451f-ac25-7e0ab3e46971 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:14.052763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c074e6-a758-451f-ac25-7e0ab3e46971 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:14.107360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3849c7-f9e4-4998-91ba-759a73c588a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:14.108106 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3849c7-f9e4-4998-91ba-759a73c588a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:14.108784 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3849c7-f9e4-4998-91ba-759a73c588a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:14.109254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3849c7-f9e4-4998-91ba-759a73c588a7 admin admin] PATH_INFO: `/v3/projects/d51a59a863ab4ba1bd03f51f0a90846e/users/0a8e9d8126734ccbb8913f4976ec76ce/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:14.113241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3849c7-f9e4-4998-91ba-759a73c588a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d51a59a863ab4ba1bd03f51f0a90846e, user_id=0a8e9d8126734ccbb8913f4976ec76ce, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:14.123065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3849c7-f9e4-4998-91ba-759a73c588a7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:14.165627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 878/1751] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:14 2022] PUT /identity/v3/projects/d51a59a863ab4ba1bd03f51f0a90846e/users/0a8e9d8126734ccbb8913f4976ec76ce/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:14.179858 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d726ad37-5e38-49b4-a191-3d60c9ef227e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:14.180436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d726ad37-5e38-49b4-a191-3d60c9ef227e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:14.180436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d726ad37-5e38-49b4-a191-3d60c9ef227e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:14.296267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d726ad37-5e38-49b4-a191-3d60c9ef227e None None] MFA Rules not processed for user `0a8e9d8126734ccbb8913f4976ec76ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:14.304254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d726ad37-5e38-49b4-a191-3d60c9ef227e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:14.583875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3849c7-f9e4-4998-91ba-759a73c588a7 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:14.586484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9e3849c7-f9e4-4998-91ba-759a73c588a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:14.609855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:14.715223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 874/1752] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:14 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 540 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:14.734957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d726ad37-5e38-49b4-a191-3d60c9ef227e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:14.737803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d726ad37-5e38-49b4-a191-3d60c9ef227e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:14.781364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3849c7-f9e4-4998-91ba-759a73c588a7 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:14.791771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:14.792501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:14.792778 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:14.793243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] PATH_INFO: `/v3/users/7c4f9efdfd2d498d8f76360bb17fd021` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:14.796756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c4f9efdfd2d498d8f76360bb17fd021)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:14.798947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:14.812333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:14.967047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c4f9efdfd2d498d8f76360bb17fd021'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:14.992969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d726ad37-5e38-49b4-a191-3d60c9ef227e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:14.996769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c4f9efdfd2d498d8f76360bb17fd021'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:14.998269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c4f9efdfd2d498d8f76360bb17fd021'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:14.999086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd4c597-319e-49a0-ac91-e01e101b241d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:14.999372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c4f9efdfd2d498d8f76360bb17fd021'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:14.999865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd4c597-319e-49a0-ac91-e01e101b241d service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:15.000220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd4c597-319e-49a0-ac91-e01e101b241d service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:15.000544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd4c597-319e-49a0-ac91-e01e101b241d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:15.017918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9dd4c597-319e-49a0-ac91-e01e101b241d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:15.030036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cad85f6f-bc04-4114-b043-be0badf5c7f7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c4f9efdfd2d498d8f76360bb17fd021'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:15.041040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd4c597-319e-49a0-ac91-e01e101b241d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:15.045653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd4c597-319e-49a0-ac91-e01e101b241d service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:15.061469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 875/1753] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:14 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:15.097072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 879/1754] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:14 2022] DELETE /identity/v3/users/7c4f9efdfd2d498d8f76360bb17fd021 => generated 0 bytes in 514 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:15.104307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3b0ebc-f27e-47c9-8472-a3c5b726d29b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:15.104856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3b0ebc-f27e-47c9-8472-a3c5b726d29b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:15.105593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3b0ebc-f27e-47c9-8472-a3c5b726d29b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:15.163967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-fa3b0ebc-f27e-47c9-8472-a3c5b726d29b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:15.165141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fa3b0ebc-f27e-47c9-8472-a3c5b726d29b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:15.329793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d4ecf2-0842-45e1-93b9-ef29e0e2ffa3 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:15.329793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d4ecf2-0842-45e1-93b9-ef29e0e2ffa3 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:15.329793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d4ecf2-0842-45e1-93b9-ef29e0e2ffa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:15.409458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 876/1755] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:30:15.556275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a7d4ecf2-0842-45e1-93b9-ef29e0e2ffa3 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:15.557562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a7d4ecf2-0842-45e1-93b9-ef29e0e2ffa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:15.747255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3b0ebc-f27e-47c9-8472-a3c5b726d29b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:15.749614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fa3b0ebc-f27e-47c9-8472-a3c5b726d29b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:15.851371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 880/1756] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:30:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 523 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:15.924791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c79fa6-4793-4bf5-b1eb-2a469f5e4a14 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:15.925255 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c79fa6-4793-4bf5-b1eb-2a469f5e4a14 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:15.925661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c79fa6-4793-4bf5-b1eb-2a469f5e4a14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:15.997757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-02c79fa6-4793-4bf5-b1eb-2a469f5e4a14 None None] MFA Rules not processed for user `97189d5c87c94be2bef7cff79c471c7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:16.006634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-02c79fa6-4793-4bf5-b1eb-2a469f5e4a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:16.011674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3b0ebc-f27e-47c9-8472-a3c5b726d29b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:16.025650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cd0d70-4b01-484d-afa0-24550fd7a1ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:16.040708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd0d70-4b01-484d-afa0-24550fd7a1ea service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:16.041021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd0d70-4b01-484d-afa0-24550fd7a1ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:16.041301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd0d70-4b01-484d-afa0-24550fd7a1ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:16.101503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8cd0d70-4b01-484d-afa0-24550fd7a1ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:16.114363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8cd0d70-4b01-484d-afa0-24550fd7a1ea service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:16.223410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 881/1757] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:15 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:16.224675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c79fa6-4793-4bf5-b1eb-2a469f5e4a14 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:16.227537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-02c79fa6-4793-4bf5-b1eb-2a469f5e4a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:16.234364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 877/1758] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:16.240439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:16.240623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cd0d70-4b01-484d-afa0-24550fd7a1ea service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:16.245048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e8cd0d70-4b01-484d-afa0-24550fd7a1ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:16.311691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cd0d70-4b01-484d-afa0-24550fd7a1ea service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:16.330446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:16.333376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:16.333680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:16.333960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] PATH_INFO: `/v3/users/c90ae5e8c0dd48e38f7caf75f214afb4` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:16.338113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c90ae5e8c0dd48e38f7caf75f214afb4)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:16.343167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:16.499104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:16.569043 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c90ae5e8c0dd48e38f7caf75f214afb4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:16.574685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c90ae5e8c0dd48e38f7caf75f214afb4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:16.630353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c90ae5e8c0dd48e38f7caf75f214afb4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:16.635594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c90ae5e8c0dd48e38f7caf75f214afb4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:16.672842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c0c440b7-cdb6-47d3-9b22-2c5d7c455b50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c90ae5e8c0dd48e38f7caf75f214afb4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:16.780655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c79fa6-4793-4bf5-b1eb-2a469f5e4a14 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:16.797869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 878/1759] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:16 2022] DELETE /identity/v3/users/c90ae5e8c0dd48e38f7caf75f214afb4 => generated 0 bytes in 563 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:16.813652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc414e86-ebeb-4fc3-8f1f-96d80a877150 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:16.817486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb53b8a-5605-41e2-8261-3d9dcdae8c81 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:16.819288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc414e86-ebeb-4fc3-8f1f-96d80a877150 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:16.819649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc414e86-ebeb-4fc3-8f1f-96d80a877150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:16.866756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb53b8a-5605-41e2-8261-3d9dcdae8c81 service swift] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:16.870972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb53b8a-5605-41e2-8261-3d9dcdae8c81 service swift] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:16.871937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb53b8a-5605-41e2-8261-3d9dcdae8c81 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:16.873849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-dc414e86-ebeb-4fc3-8f1f-96d80a877150 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:16.880379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-dc414e86-ebeb-4fc3-8f1f-96d80a877150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:16.885823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ecb53b8a-5605-41e2-8261-3d9dcdae8c81 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:16.913465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb53b8a-5605-41e2-8261-3d9dcdae8c81 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:16.917352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb53b8a-5605-41e2-8261-3d9dcdae8c81 service swift] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:16.946847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 882/1760] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:30:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 722 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:30:17.136843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 879/1761] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:17.390223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb53b8a-5605-41e2-8261-3d9dcdae8c81 service swift] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:17.398958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ecb53b8a-5605-41e2-8261-3d9dcdae8c81 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:17.443131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb53b8a-5605-41e2-8261-3d9dcdae8c81 service swift] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:17.653704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b0b1ea-d32a-4961-abae-69bea22659e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:17.654303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b0b1ea-d32a-4961-abae-69bea22659e0 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:17.654588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b0b1ea-d32a-4961-abae-69bea22659e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:17.654858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b0b1ea-d32a-4961-abae-69bea22659e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:17.674554 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b0b1ea-d32a-4961-abae-69bea22659e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:17.676176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b0b1ea-d32a-4961-abae-69bea22659e0 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:17.702336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 883/1762] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:19.405286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc414e86-ebeb-4fc3-8f1f-96d80a877150 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:19.407089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-dc414e86-ebeb-4fc3-8f1f-96d80a877150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:19.433314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc414e86-ebeb-4fc3-8f1f-96d80a877150 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:19.447418 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:19.452624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:19.453745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:19.456837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] PATH_INFO: `/v3/projects/3db54de0d87e4f9ea30a70cfddfd595e` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:19.459929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3db54de0d87e4f9ea30a70cfddfd595e)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:19.465425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:19.585789 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3db54de0d87e4f9ea30a70cfddfd595e', 'request_id': 'req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:19.599112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3db54de0d87e4f9ea30a70cfddfd595e', 'request_id': 'req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:19.622472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3db54de0d87e4f9ea30a70cfddfd595e', 'request_id': 'req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:19.657543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] The token cache is being invalidate because project 3db54de0d87e4f9ea30a70cfddfd595e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:19.663738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:19.663738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1be1394a-f8c9-4629-be3f-eb482e7a5cc0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:19.666067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 880/1763] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:19 2022] DELETE /identity/v3/projects/3db54de0d87e4f9ea30a70cfddfd595e => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:19.670594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b0b1ea-d32a-4961-abae-69bea22659e0 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:19.687023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b0b1ea-d32a-4961-abae-69bea22659e0 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:19.769668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1db96aa-d296-4378-914d-67634e970846 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:19.770059 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1db96aa-d296-4378-914d-67634e970846 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:19.770477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1db96aa-d296-4378-914d-67634e970846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:19.834767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d1db96aa-d296-4378-914d-67634e970846 None None] MFA Rules not processed for user `c9a79828d83e4535bb46631b52b7d7ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:19.838437 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d1db96aa-d296-4378-914d-67634e970846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:19.856549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:19.857218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:19.857503 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:19.857772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] PATH_INFO: `/v3/users/416ef1fbdca54e80ac09b6648cda5642` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:19.862561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=416ef1fbdca54e80ac09b6648cda5642)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:19.864975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:19.884160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:19.905072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:19.993300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '416ef1fbdca54e80ac09b6648cda5642'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:20.008278 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 881/1764] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:19 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:30:20.014165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1db96aa-d296-4378-914d-67634e970846 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:20.019100 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d1db96aa-d296-4378-914d-67634e970846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:20.021751 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '416ef1fbdca54e80ac09b6648cda5642'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:20.022566 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '416ef1fbdca54e80ac09b6648cda5642'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:20.023584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '416ef1fbdca54e80ac09b6648cda5642'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:20.048511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1db96aa-d296-4378-914d-67634e970846 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:20.066312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99699666-ced5-4602-8a31-937e5bdb6f48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:20.067110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99699666-ced5-4602-8a31-937e5bdb6f48 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:20.067522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99699666-ced5-4602-8a31-937e5bdb6f48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:20.068270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99699666-ced5-4602-8a31-937e5bdb6f48 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:20.070883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '416ef1fbdca54e80ac09b6648cda5642'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:20.079009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99699666-ced5-4602-8a31-937e5bdb6f48 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:20.082648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99699666-ced5-4602-8a31-937e5bdb6f48 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:20.154307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 884/1765] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:19 2022] DELETE /identity/v3/users/416ef1fbdca54e80ac09b6648cda5642 => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:20.161708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 882/1766] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:20 2022] POST /identity/v3/projects => generated 373 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:30:20.167064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:20.169799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:20.382461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81512e13-28b7-45e1-b325-2a07572d1bb8 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:20.584209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc812a76-021c-47ac-bac0-5c1581bf2ede admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:20.585473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc812a76-021c-47ac-bac0-5c1581bf2ede admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:20.587472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc812a76-021c-47ac-bac0-5c1581bf2ede admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:20.587777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc812a76-021c-47ac-bac0-5c1581bf2ede admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:20.590611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc812a76-021c-47ac-bac0-5c1581bf2ede admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:20.592794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc812a76-021c-47ac-bac0-5c1581bf2ede admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:20.655649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 885/1767] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:20 2022] POST /identity/v3/users => generated 293 bytes in 489 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:20.659997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99699666-ced5-4602-8a31-937e5bdb6f48 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:20.684807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99699666-ced5-4602-8a31-937e5bdb6f48 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:20.712618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf52e82-2fca-4137-9467-b9fef0d6af03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:20.713362 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf52e82-2fca-4137-9467-b9fef0d6af03 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:20.713735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf52e82-2fca-4137-9467-b9fef0d6af03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:20.714017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf52e82-2fca-4137-9467-b9fef0d6af03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:20.714757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf52e82-2fca-4137-9467-b9fef0d6af03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:20.716350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf52e82-2fca-4137-9467-b9fef0d6af03 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:20.770027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 883/1768] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:20 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:20.774598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc812a76-021c-47ac-bac0-5c1581bf2ede admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:20.793927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc812a76-021c-47ac-bac0-5c1581bf2ede admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:20.817638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cba677-e80c-4c7c-ae0a-489a648ec3f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:20.819093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cba677-e80c-4c7c-ae0a-489a648ec3f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:20.820333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cba677-e80c-4c7c-ae0a-489a648ec3f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:20.820615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cba677-e80c-4c7c-ae0a-489a648ec3f3 admin admin] PATH_INFO: `/v3/projects/b9231b54e2b447bdba74d6282decf896/users/a9040b0c066142438efa9a9e603e4ed0/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:20.864103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cba677-e80c-4c7c-ae0a-489a648ec3f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9231b54e2b447bdba74d6282decf896, user_id=a9040b0c066142438efa9a9e603e4ed0, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:20.869579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cba677-e80c-4c7c-ae0a-489a648ec3f3 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:20.916801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 886/1769] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:20 2022] PUT /identity/v3/projects/b9231b54e2b447bdba74d6282decf896/users/a9040b0c066142438efa9a9e603e4ed0/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:20.922170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf52e82-2fca-4137-9467-b9fef0d6af03 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:20.935296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf52e82-2fca-4137-9467-b9fef0d6af03 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:21.327656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53113f3-3a02-4138-9b16-7977a957ec41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:21.328371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53113f3-3a02-4138-9b16-7977a957ec41 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:21.328660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53113f3-3a02-4138-9b16-7977a957ec41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:21.328939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53113f3-3a02-4138-9b16-7977a957ec41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:21.329661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53113f3-3a02-4138-9b16-7977a957ec41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:21.355611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53113f3-3a02-4138-9b16-7977a957ec41 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:21.396606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 884/1770] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:20 2022] GET /identity/v3/roles => generated 1420 bytes in 475 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:21.401179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cba677-e80c-4c7c-ae0a-489a648ec3f3 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:21.426073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cba677-e80c-4c7c-ae0a-489a648ec3f3 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:21.444053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f01241-931d-40eb-95fc-e7a0dc65f8f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:21.444765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f01241-931d-40eb-95fc-e7a0dc65f8f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:21.445030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f01241-931d-40eb-95fc-e7a0dc65f8f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:21.445306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f01241-931d-40eb-95fc-e7a0dc65f8f7 admin admin] PATH_INFO: `/v3/projects/b9231b54e2b447bdba74d6282decf896/users/a9040b0c066142438efa9a9e603e4ed0/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:21.449883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f01241-931d-40eb-95fc-e7a0dc65f8f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9231b54e2b447bdba74d6282decf896, user_id=a9040b0c066142438efa9a9e603e4ed0, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:21.457058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f01241-931d-40eb-95fc-e7a0dc65f8f7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:21.462063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=9,keystone:SELECT=86,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:21.494178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 887/1771] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:21 2022] PUT /identity/v3/projects/b9231b54e2b447bdba74d6282decf896/users/a9040b0c066142438efa9a9e603e4ed0/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:30:21.578855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53113f3-3a02-4138-9b16-7977a957ec41 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:21.581585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d53113f3-3a02-4138-9b16-7977a957ec41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:21.651430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53113f3-3a02-4138-9b16-7977a957ec41 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:21.902824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:21.903589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:21.905269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:21.905755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] PATH_INFO: `/v3/projects/bb2a6edb2dc34dd29701c9d74541ed08` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:21.909375 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb2a6edb2dc34dd29701c9d74541ed08)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:21.911598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:22.021386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:22.054685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb2a6edb2dc34dd29701c9d74541ed08', 'request_id': 'req-e10b2e79-449c-4107-8a95-d7cc0fe6391f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:22.069925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb2a6edb2dc34dd29701c9d74541ed08', 'request_id': 'req-e10b2e79-449c-4107-8a95-d7cc0fe6391f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:22.090412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb2a6edb2dc34dd29701c9d74541ed08', 'request_id': 'req-e10b2e79-449c-4107-8a95-d7cc0fe6391f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:22.118334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] The token cache is being invalidate because project bb2a6edb2dc34dd29701c9d74541ed08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:22.121063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:22.123822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e10b2e79-449c-4107-8a95-d7cc0fe6391f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:22.131236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 885/1772] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:21 2022] DELETE /identity/v3/projects/bb2a6edb2dc34dd29701c9d74541ed08 => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:22.135179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f01241-931d-40eb-95fc-e7a0dc65f8f7 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:22.138045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-91f01241-931d-40eb-95fc-e7a0dc65f8f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:22.179541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f01241-931d-40eb-95fc-e7a0dc65f8f7 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:22.446809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:22.447608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:22.448345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:22.448345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] PATH_INFO: `/v3/users/0a8e9d8126734ccbb8913f4976ec76ce` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:22.450477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a8e9d8126734ccbb8913f4976ec76ce)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:22.452916 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:22.553031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2289180-cb74-44f1-8676-31795769cdb7 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:22.553830 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2289180-cb74-44f1-8676-31795769cdb7 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:22.554350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2289180-cb74-44f1-8676-31795769cdb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:22.659538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e2289180-cb74-44f1-8676-31795769cdb7 None None] MFA Rules not processed for user `515e0f8a3a234deba5f909271b303abb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:22.664628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e2289180-cb74-44f1-8676-31795769cdb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:22.665356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a8e9d8126734ccbb8913f4976ec76ce'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:22.685793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a8e9d8126734ccbb8913f4976ec76ce'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:22.686781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a8e9d8126734ccbb8913f4976ec76ce'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:22.688797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a8e9d8126734ccbb8913f4976ec76ce'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:22.709012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a8e9d8126734ccbb8913f4976ec76ce'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:22.762125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 888/1773] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:22 2022] DELETE /identity/v3/users/0a8e9d8126734ccbb8913f4976ec76ce => generated 0 bytes in 627 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:22.994933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 886/1774] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:22 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 443 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:23.207384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:23.209478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:23.480436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71c1365-f099-4816-a66d-b0a43bb4d7e9 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:23.483079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa4ce3f-27cf-4534-94a2-20fcb951991e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:23.483368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa4ce3f-27cf-4534-94a2-20fcb951991e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:23.484602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa4ce3f-27cf-4534-94a2-20fcb951991e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:23.522288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:23.523505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:23.527091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:23.529140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] PATH_INFO: `/v3/projects/09c41d0a733a421280da4085ac2b3825` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:23.531805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] RBAC: Authorizing `identity:delete_project(project_id=09c41d0a733a421280da4085ac2b3825)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:23.546463 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:23.555972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7aa4ce3f-27cf-4534-94a2-20fcb951991e None None] MFA Rules not processed for user `171e916244804add8f18b44b77b03d72`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:23.562777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7aa4ce3f-27cf-4534-94a2-20fcb951991e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:23.616171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:23.690665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09c41d0a733a421280da4085ac2b3825', 'request_id': 'req-3f1f4161-3c23-4b73-b7d2-0895faaa774c'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:23.703393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09c41d0a733a421280da4085ac2b3825', 'request_id': 'req-3f1f4161-3c23-4b73-b7d2-0895faaa774c'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:23.721605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09c41d0a733a421280da4085ac2b3825', 'request_id': 'req-3f1f4161-3c23-4b73-b7d2-0895faaa774c'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:23.731500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 887/1775] 10.222.0.42 () {58 vars in 1015 bytes} [Tue Sep 13 15:30:23 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:23.773432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] The token cache is being invalidate because project 09c41d0a733a421280da4085ac2b3825 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:23.773949 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:23.775508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3f1f4161-3c23-4b73-b7d2-0895faaa774c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:23.778302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 889/1776] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:23 2022] DELETE /identity/v3/projects/09c41d0a733a421280da4085ac2b3825 => generated 0 bytes in 572 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:23.782913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b059e54-3041-4dec-948c-3ce3befb02dd None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:23.783197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b059e54-3041-4dec-948c-3ce3befb02dd None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:23.783616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b059e54-3041-4dec-948c-3ce3befb02dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:23.811854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-5b059e54-3041-4dec-948c-3ce3befb02dd None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:23.814517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-5b059e54-3041-4dec-948c-3ce3befb02dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:24.006123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 888/1777] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:24.014484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff76b06-2a4e-4617-8c8a-883218717ee6 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:24.014769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff76b06-2a4e-4617-8c8a-883218717ee6 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:24.015060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff76b06-2a4e-4617-8c8a-883218717ee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:24.037728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-5ff76b06-2a4e-4617-8c8a-883218717ee6 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:24.039005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5ff76b06-2a4e-4617-8c8a-883218717ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:24.058850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b059e54-3041-4dec-948c-3ce3befb02dd None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:24.060086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 890/1778] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:24.073810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b059e54-3041-4dec-948c-3ce3befb02dd None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:24.098185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43386da0-d7e4-4c35-832c-5eb867899f78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:24.103373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43386da0-d7e4-4c35-832c-5eb867899f78 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:24.103655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43386da0-d7e4-4c35-832c-5eb867899f78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:24.104009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43386da0-d7e4-4c35-832c-5eb867899f78 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:24.105083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43386da0-d7e4-4c35-832c-5eb867899f78 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:30:24.105585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43386da0-d7e4-4c35-832c-5eb867899f78 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:24.107365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43386da0-d7e4-4c35-832c-5eb867899f78 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:24.115813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:24.131331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 889/1779] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:30:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:30:24.140309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff76b06-2a4e-4617-8c8a-883218717ee6 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:24.151136 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff76b06-2a4e-4617-8c8a-883218717ee6 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:24.183628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a77a07-e5bb-4c1c-ab7f-58bd295fc1a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:24.185252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a77a07-e5bb-4c1c-ab7f-58bd295fc1a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:24.186542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a77a07-e5bb-4c1c-ab7f-58bd295fc1a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:24.186858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a77a07-e5bb-4c1c-ab7f-58bd295fc1a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:24.193783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a77a07-e5bb-4c1c-ab7f-58bd295fc1a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:24.200411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a77a07-e5bb-4c1c-ab7f-58bd295fc1a9 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:24.301639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 891/1780] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:24 2022] POST /identity/v3/projects => generated 389 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:24.303121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43386da0-d7e4-4c35-832c-5eb867899f78 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:24.321820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43386da0-d7e4-4c35-832c-5eb867899f78 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:24.326799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a77a07-e5bb-4c1c-ab7f-58bd295fc1a9 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:24.329477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-97a77a07-e5bb-4c1c-ab7f-58bd295fc1a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:24.551172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a77a07-e5bb-4c1c-ab7f-58bd295fc1a9 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:24.561432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94af1f3d-5048-409f-adbc-08f54d2ad87f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:24.562079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94af1f3d-5048-409f-adbc-08f54d2ad87f admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:24.562384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94af1f3d-5048-409f-adbc-08f54d2ad87f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:24.562654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94af1f3d-5048-409f-adbc-08f54d2ad87f admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:24.562964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96b4a38-e1e2-4dc7-b729-4da5a70be7ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:24.563297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94af1f3d-5048-409f-adbc-08f54d2ad87f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:24.565300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94af1f3d-5048-409f-adbc-08f54d2ad87f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:24.565824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96b4a38-e1e2-4dc7-b729-4da5a70be7ef service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:24.566696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96b4a38-e1e2-4dc7-b729-4da5a70be7ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:24.567196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96b4a38-e1e2-4dc7-b729-4da5a70be7ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:24.570006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b96b4a38-e1e2-4dc7-b729-4da5a70be7ef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:24.584685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96b4a38-e1e2-4dc7-b729-4da5a70be7ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:24.587394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96b4a38-e1e2-4dc7-b729-4da5a70be7ef service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:24.604687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 892/1781] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:24 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:30:24.606169 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96b4a38-e1e2-4dc7-b729-4da5a70be7ef service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:24.638578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96b4a38-e1e2-4dc7-b729-4da5a70be7ef service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:24.660397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d80c5e-7982-4f4d-bb40-c8c4f1d5fd7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:24.662063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d80c5e-7982-4f4d-bb40-c8c4f1d5fd7a service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:24.662569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d80c5e-7982-4f4d-bb40-c8c4f1d5fd7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:24.663085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d80c5e-7982-4f4d-bb40-c8c4f1d5fd7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:24.674855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d80c5e-7982-4f4d-bb40-c8c4f1d5fd7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:24.677206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d80c5e-7982-4f4d-bb40-c8c4f1d5fd7a service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:24.687953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 893/1782] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:24 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:24.689470 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 890/1783] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:24 2022] POST /identity/v3/users => generated 295 bytes in 387 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:24.695165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d80c5e-7982-4f4d-bb40-c8c4f1d5fd7a service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:24.715227 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d80c5e-7982-4f4d-bb40-c8c4f1d5fd7a service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:24.721514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627c8622-e403-4912-9835-33abe37490c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:24.722353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627c8622-e403-4912-9835-33abe37490c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:24.723808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627c8622-e403-4912-9835-33abe37490c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:24.724184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627c8622-e403-4912-9835-33abe37490c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:24.724950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627c8622-e403-4912-9835-33abe37490c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:24.734147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627c8622-e403-4912-9835-33abe37490c3 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:24.766409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 894/1784] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:24 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:24.767772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94af1f3d-5048-409f-adbc-08f54d2ad87f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:24.793263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94af1f3d-5048-409f-adbc-08f54d2ad87f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:24.825256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a5ca16-2a7c-46b0-9bbc-e40eb4eab29f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:24.826287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a5ca16-2a7c-46b0-9bbc-e40eb4eab29f admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:24.829028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a5ca16-2a7c-46b0-9bbc-e40eb4eab29f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:24.829580 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a5ca16-2a7c-46b0-9bbc-e40eb4eab29f admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:24.830533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a5ca16-2a7c-46b0-9bbc-e40eb4eab29f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:24.836914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a5ca16-2a7c-46b0-9bbc-e40eb4eab29f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:24.912573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 891/1785] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:24 2022] GET /identity/v3/roles => generated 1420 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:24.916032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627c8622-e403-4912-9835-33abe37490c3 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:24.966351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627c8622-e403-4912-9835-33abe37490c3 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:24.980336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2fd924-bc81-4ecf-a76d-62f7da7eb22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:24.981084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2fd924-bc81-4ecf-a76d-62f7da7eb22d admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:24.981546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2fd924-bc81-4ecf-a76d-62f7da7eb22d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:24.981834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2fd924-bc81-4ecf-a76d-62f7da7eb22d admin admin] PATH_INFO: `/v3/projects/146a77f3ce4c4463ae8e9af9731710c7/users/1899d9dbb96e43f0b1fa5f0cac66352c/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:25.087805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2fd924-bc81-4ecf-a76d-62f7da7eb22d admin admin] RBAC: Authorizing `identity:create_grant(project_id=146a77f3ce4c4463ae8e9af9731710c7, user_id=1899d9dbb96e43f0b1fa5f0cac66352c, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:25.095899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2fd924-bc81-4ecf-a76d-62f7da7eb22d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:25.153676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 895/1786] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:24 2022] PUT /identity/v3/projects/146a77f3ce4c4463ae8e9af9731710c7/users/1899d9dbb96e43f0b1fa5f0cac66352c/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:25.153902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cc4ccd-aa8b-4f83-b0df-cc78e5f14fdb None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:25.154421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cc4ccd-aa8b-4f83-b0df-cc78e5f14fdb None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:25.154421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cc4ccd-aa8b-4f83-b0df-cc78e5f14fdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:25.250922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-f4cc4ccd-aa8b-4f83-b0df-cc78e5f14fdb None None] MFA Rules not processed for user `1899d9dbb96e43f0b1fa5f0cac66352c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:25.252955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f4cc4ccd-aa8b-4f83-b0df-cc78e5f14fdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:25.441056 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 892/1787] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:25 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:25.445817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2fd924-bc81-4ecf-a76d-62f7da7eb22d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:25.535745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:25.622681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2fd924-bc81-4ecf-a76d-62f7da7eb22d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:25.651483 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31e78bc-c1a3-488f-8e82-89fe4e967aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:25.652384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31e78bc-c1a3-488f-8e82-89fe4e967aea admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:25.652668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31e78bc-c1a3-488f-8e82-89fe4e967aea admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:25.652942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31e78bc-c1a3-488f-8e82-89fe4e967aea admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:25.653596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31e78bc-c1a3-488f-8e82-89fe4e967aea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:25.659728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31e78bc-c1a3-488f-8e82-89fe4e967aea admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:25.720505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 896/1788] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:25 2022] POST /identity/v3/projects => generated 391 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:25.726277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cc4ccd-aa8b-4f83-b0df-cc78e5f14fdb None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:25.746620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cc4ccd-aa8b-4f83-b0df-cc78e5f14fdb None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:25.994558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606fdafa-dbf7-4620-800e-d9626fa6c5ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:25.997159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606fdafa-dbf7-4620-800e-d9626fa6c5ce admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:25.998323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606fdafa-dbf7-4620-800e-d9626fa6c5ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:25.999862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606fdafa-dbf7-4620-800e-d9626fa6c5ce admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:26.002735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606fdafa-dbf7-4620-800e-d9626fa6c5ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:26.014003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606fdafa-dbf7-4620-800e-d9626fa6c5ce admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:26.076052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:26.099349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 893/1789] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:25 2022] POST /identity/v3/users => generated 302 bytes in 375 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:26.103653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31e78bc-c1a3-488f-8e82-89fe4e967aea admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:26.121828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31e78bc-c1a3-488f-8e82-89fe4e967aea admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:26.149853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8991f2fe-7bbd-4185-ba57-ef843a7b5b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:26.150551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8991f2fe-7bbd-4185-ba57-ef843a7b5b56 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:26.150881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8991f2fe-7bbd-4185-ba57-ef843a7b5b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:26.151108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8991f2fe-7bbd-4185-ba57-ef843a7b5b56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:26.152162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8991f2fe-7bbd-4185-ba57-ef843a7b5b56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:26.159265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8991f2fe-7bbd-4185-ba57-ef843a7b5b56 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:26.188247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606fdafa-dbf7-4620-800e-d9626fa6c5ce admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:26.197300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 897/1790] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:26 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:26.204728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606fdafa-dbf7-4620-800e-d9626fa6c5ce admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:26.218807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92386e2b-e53e-44a0-b57c-d29e18ea25aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:26.222057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92386e2b-e53e-44a0-b57c-d29e18ea25aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:26.223967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92386e2b-e53e-44a0-b57c-d29e18ea25aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:26.224249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92386e2b-e53e-44a0-b57c-d29e18ea25aa admin admin] PATH_INFO: `/v3/projects/ea0448df78df427dbfac27521be7a626/users/528cfa4a906347e79698f3dcb89ee645/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:26.269174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92386e2b-e53e-44a0-b57c-d29e18ea25aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea0448df78df427dbfac27521be7a626, user_id=528cfa4a906347e79698f3dcb89ee645, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:26.278996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92386e2b-e53e-44a0-b57c-d29e18ea25aa admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:26.317201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 894/1791] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:26 2022] PUT /identity/v3/projects/ea0448df78df427dbfac27521be7a626/users/528cfa4a906347e79698f3dcb89ee645/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:26.319524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8991f2fe-7bbd-4185-ba57-ef843a7b5b56 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:26.333666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8991f2fe-7bbd-4185-ba57-ef843a7b5b56 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:26.535424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d8d14d-7cc2-4f14-931e-c5106e229dec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:26.537276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d8d14d-7cc2-4f14-931e-c5106e229dec admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:26.543986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d8d14d-7cc2-4f14-931e-c5106e229dec admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:26.544297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d8d14d-7cc2-4f14-931e-c5106e229dec admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:26.545014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d8d14d-7cc2-4f14-931e-c5106e229dec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:26.546829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d8d14d-7cc2-4f14-931e-c5106e229dec admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:26.600137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 898/1792] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:26 2022] GET /identity/v3/roles => generated 1420 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:26.604392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92386e2b-e53e-44a0-b57c-d29e18ea25aa admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:26.622986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92386e2b-e53e-44a0-b57c-d29e18ea25aa admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:26.652718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6bb659-a10a-41e7-b660-151d0103b5fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:26.653419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6bb659-a10a-41e7-b660-151d0103b5fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:26.653702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6bb659-a10a-41e7-b660-151d0103b5fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:26.653977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6bb659-a10a-41e7-b660-151d0103b5fa admin admin] PATH_INFO: `/v3/projects/ea0448df78df427dbfac27521be7a626/users/528cfa4a906347e79698f3dcb89ee645/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:26.656864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6bb659-a10a-41e7-b660-151d0103b5fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea0448df78df427dbfac27521be7a626, user_id=528cfa4a906347e79698f3dcb89ee645, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:26.662003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6bb659-a10a-41e7-b660-151d0103b5fa admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:26.703854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 895/1793] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:26 2022] PUT /identity/v3/projects/ea0448df78df427dbfac27521be7a626/users/528cfa4a906347e79698f3dcb89ee645/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:26.726539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7d0596-3f40-4c73-8e02-93287997198c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:26.726822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7d0596-3f40-4c73-8e02-93287997198c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:26.727094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7d0596-3f40-4c73-8e02-93287997198c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:26.787430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-9f7d0596-3f40-4c73-8e02-93287997198c None None] MFA Rules not processed for user `528cfa4a906347e79698f3dcb89ee645`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:26.788681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9f7d0596-3f40-4c73-8e02-93287997198c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:26.996470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 899/1794] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:26 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:27.011114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6bb659-a10a-41e7-b660-151d0103b5fa admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:27.014796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0f6bb659-a10a-41e7-b660-151d0103b5fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:27.239445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6bb659-a10a-41e7-b660-151d0103b5fa admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:27.247007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c361e4e-49e0-4086-aadd-b664dd94089b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:27.247568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c361e4e-49e0-4086-aadd-b664dd94089b service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:27.247839 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c361e4e-49e0-4086-aadd-b664dd94089b service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:27.248103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c361e4e-49e0-4086-aadd-b664dd94089b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:27.270845 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c361e4e-49e0-4086-aadd-b664dd94089b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:27.274651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c361e4e-49e0-4086-aadd-b664dd94089b service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:27.291936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 896/1795] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:27 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:27.423297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7d0596-3f40-4c73-8e02-93287997198c None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:27.425276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9f7d0596-3f40-4c73-8e02-93287997198c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:27.493624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c361e4e-49e0-4086-aadd-b664dd94089b service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:27.498505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1c361e4e-49e0-4086-aadd-b664dd94089b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:27.569128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:27.595542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c361e4e-49e0-4086-aadd-b664dd94089b service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:27.636221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cbff89-802e-4f4b-a4aa-64288ecf088f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:27.638162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cbff89-802e-4f4b-a4aa-64288ecf088f service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:27.643134 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cbff89-802e-4f4b-a4aa-64288ecf088f service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:27.644483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cbff89-802e-4f4b-a4aa-64288ecf088f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:27.725555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7d0596-3f40-4c73-8e02-93287997198c None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:27.740070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:27.740752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:27.741033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:27.741308 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] PATH_INFO: `/v3/projects/d51a59a863ab4ba1bd03f51f0a90846e` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:27.742608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d51a59a863ab4ba1bd03f51f0a90846e)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:27.745773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:27.902024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd51a59a863ab4ba1bd03f51f0a90846e', 'request_id': 'req-cde8b0af-b963-4390-973c-72f77c37c176'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:27.924687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd51a59a863ab4ba1bd03f51f0a90846e', 'request_id': 'req-cde8b0af-b963-4390-973c-72f77c37c176'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:27.944130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd51a59a863ab4ba1bd03f51f0a90846e', 'request_id': 'req-cde8b0af-b963-4390-973c-72f77c37c176'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:27.991090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] The token cache is being invalidate because project d51a59a863ab4ba1bd03f51f0a90846e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:27.997882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:28.004442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-cde8b0af-b963-4390-973c-72f77c37c176 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:28.014826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 900/1796] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:27 2022] DELETE /identity/v3/projects/d51a59a863ab4ba1bd03f51f0a90846e => generated 0 bytes in 590 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:28.023752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5416be-670f-4a4a-8511-0feaa5234ff2 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:28.024325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5416be-670f-4a4a-8511-0feaa5234ff2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:28.024695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5416be-670f-4a4a-8511-0feaa5234ff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:28.111879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-fb5416be-670f-4a4a-8511-0feaa5234ff2 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:28.113272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-fb5416be-670f-4a4a-8511-0feaa5234ff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:28.121241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cbff89-802e-4f4b-a4aa-64288ecf088f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:28.123253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cbff89-802e-4f4b-a4aa-64288ecf088f service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:28.148928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 897/1797] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:27 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 656 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:28.345877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 901/1798] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:30:28.355224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed3d15b-77a4-436f-bff1-33e46a3c2453 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:28.355755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed3d15b-77a4-436f-bff1-33e46a3c2453 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:28.356368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed3d15b-77a4-436f-bff1-33e46a3c2453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:28.396413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8ed3d15b-77a4-436f-bff1-33e46a3c2453 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:28.398080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8ed3d15b-77a4-436f-bff1-33e46a3c2453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:28.420698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 898/1799] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:28.425975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5416be-670f-4a4a-8511-0feaa5234ff2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:28.444915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5416be-670f-4a4a-8511-0feaa5234ff2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:28.459091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb68868-b181-4fd6-aefb-f286b3cb509c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:28.460345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb68868-b181-4fd6-aefb-f286b3cb509c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:28.460629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb68868-b181-4fd6-aefb-f286b3cb509c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:28.461232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb68868-b181-4fd6-aefb-f286b3cb509c admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:28.462044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb68868-b181-4fd6-aefb-f286b3cb509c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:30:28.462513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb68868-b181-4fd6-aefb-f286b3cb509c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:28.465007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb68868-b181-4fd6-aefb-f286b3cb509c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:28.494417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 902/1800] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:30:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:30:28.495697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed3d15b-77a4-436f-bff1-33e46a3c2453 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:28.505910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed3d15b-77a4-436f-bff1-33e46a3c2453 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:28.523860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3c88b4-d0de-490f-9b7d-ea9639a60d9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:28.525237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3c88b4-d0de-490f-9b7d-ea9639a60d9b admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:28.525625 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3c88b4-d0de-490f-9b7d-ea9639a60d9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:28.526071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3c88b4-d0de-490f-9b7d-ea9639a60d9b admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:28.526874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3c88b4-d0de-490f-9b7d-ea9639a60d9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:28.528985 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3c88b4-d0de-490f-9b7d-ea9639a60d9b admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:28.593344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 899/1801] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:28 2022] POST /identity/v3/projects => generated 365 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:28.599366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb68868-b181-4fd6-aefb-f286b3cb509c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:28.614452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb68868-b181-4fd6-aefb-f286b3cb509c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:28.783674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:28.852217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce7fdf1-3537-4e26-b5c6-0f1bb8bf8103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:28.852865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce7fdf1-3537-4e26-b5c6-0f1bb8bf8103 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:28.853146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce7fdf1-3537-4e26-b5c6-0f1bb8bf8103 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:28.853424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce7fdf1-3537-4e26-b5c6-0f1bb8bf8103 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:28.857226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce7fdf1-3537-4e26-b5c6-0f1bb8bf8103 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:28.859337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce7fdf1-3537-4e26-b5c6-0f1bb8bf8103 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:28.985499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 903/1802] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:28 2022] POST /identity/v3/users => generated 283 bytes in 387 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:28.989661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3c88b4-d0de-490f-9b7d-ea9639a60d9b admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:29.012321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3c88b4-d0de-490f-9b7d-ea9639a60d9b admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:29.037015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052f0bec-a811-49ca-b4f8-b78256058ece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:29.042053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052f0bec-a811-49ca-b4f8-b78256058ece admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:29.042354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052f0bec-a811-49ca-b4f8-b78256058ece admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:29.045619 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052f0bec-a811-49ca-b4f8-b78256058ece admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:29.050671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052f0bec-a811-49ca-b4f8-b78256058ece admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:29.064792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052f0bec-a811-49ca-b4f8-b78256058ece admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:29.110871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 900/1803] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:28 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:29.114300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce7fdf1-3537-4e26-b5c6-0f1bb8bf8103 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:29.151563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce7fdf1-3537-4e26-b5c6-0f1bb8bf8103 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:29.175667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55050cc-2e87-454f-9263-a4300cbf820a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:29.181309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55050cc-2e87-454f-9263-a4300cbf820a admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:29.181626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55050cc-2e87-454f-9263-a4300cbf820a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:29.181879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55050cc-2e87-454f-9263-a4300cbf820a admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:29.182646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55050cc-2e87-454f-9263-a4300cbf820a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:29.194144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55050cc-2e87-454f-9263-a4300cbf820a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:29.292663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 904/1804] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:29 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:29.293960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052f0bec-a811-49ca-b4f8-b78256058ece admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:29.323046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052f0bec-a811-49ca-b4f8-b78256058ece admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:29.350560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175709a1-a8f5-4cc9-9337-5e8cca58e508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:29.354759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175709a1-a8f5-4cc9-9337-5e8cca58e508 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:29.356621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175709a1-a8f5-4cc9-9337-5e8cca58e508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:29.357569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175709a1-a8f5-4cc9-9337-5e8cca58e508 admin admin] PATH_INFO: `/v3/projects/e6530f095c314ca18ac654c146d384f7/users/02cfcbe4f42140659fb5a47230054fb6/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:29.439054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175709a1-a8f5-4cc9-9337-5e8cca58e508 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6530f095c314ca18ac654c146d384f7, user_id=02cfcbe4f42140659fb5a47230054fb6, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:29.458987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175709a1-a8f5-4cc9-9337-5e8cca58e508 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:29.469156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:29.513882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 901/1805] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:29 2022] PUT /identity/v3/projects/e6530f095c314ca18ac654c146d384f7/users/02cfcbe4f42140659fb5a47230054fb6/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:29.527311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052a6b68-96be-4377-b66c-376aa56408d8 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:29.527716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052a6b68-96be-4377-b66c-376aa56408d8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:29.528003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052a6b68-96be-4377-b66c-376aa56408d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:29.666303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-052a6b68-96be-4377-b66c-376aa56408d8 None None] MFA Rules not processed for user `02cfcbe4f42140659fb5a47230054fb6`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:29.668378 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-052a6b68-96be-4377-b66c-376aa56408d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:29.935553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 905/1806] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:29 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 424 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:30:29.939612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175709a1-a8f5-4cc9-9337-5e8cca58e508 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:30.133377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175709a1-a8f5-4cc9-9337-5e8cca58e508 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:30.172745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b9b5c-8b65-462e-9886-75bc4935902e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:30.173794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b9b5c-8b65-462e-9886-75bc4935902e admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:30.174699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b9b5c-8b65-462e-9886-75bc4935902e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:30.177860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748b9b5c-8b65-462e-9886-75bc4935902e admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:30.178933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748b9b5c-8b65-462e-9886-75bc4935902e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:30.189703 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748b9b5c-8b65-462e-9886-75bc4935902e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:30.197095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d5d2f6-4038-4054-8e3f-a00622d01960 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:30.197386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d5d2f6-4038-4054-8e3f-a00622d01960 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:30.197700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d5d2f6-4038-4054-8e3f-a00622d01960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:30.302781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d6d5d2f6-4038-4054-8e3f-a00622d01960 None None] MFA Rules not processed for user `a9040b0c066142438efa9a9e603e4ed0`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:30.303958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d6d5d2f6-4038-4054-8e3f-a00622d01960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:30.337731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 902/1807] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:29 2022] POST /identity/v3/projects => generated 363 bytes in 396 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:30:30.339549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b9b5c-8b65-462e-9886-75bc4935902e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:30.515383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 906/1808] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:30 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:30.556474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d5d2f6-4038-4054-8e3f-a00622d01960 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:30.558346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d6d5d2f6-4038-4054-8e3f-a00622d01960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:30.640358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748b9b5c-8b65-462e-9886-75bc4935902e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:30.662562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eac671-9d32-4323-84c7-6758d9d57ce5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:30.664545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eac671-9d32-4323-84c7-6758d9d57ce5 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:30.665992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eac671-9d32-4323-84c7-6758d9d57ce5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:30.666500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eac671-9d32-4323-84c7-6758d9d57ce5 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:30.668175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42eac671-9d32-4323-84c7-6758d9d57ce5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:30.673378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42eac671-9d32-4323-84c7-6758d9d57ce5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:30.745634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 903/1809] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:30 2022] POST /identity/v3/users => generated 288 bytes in 406 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:30.746691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eac671-9d32-4323-84c7-6758d9d57ce5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:30.765213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eac671-9d32-4323-84c7-6758d9d57ce5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:30.772810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:30.777147 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dcd96b-8910-4985-8c1e-ecc334f20d78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:30.777847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dcd96b-8910-4985-8c1e-ecc334f20d78 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:30.778144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dcd96b-8910-4985-8c1e-ecc334f20d78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:30.778477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dcd96b-8910-4985-8c1e-ecc334f20d78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:30.779144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0dcd96b-8910-4985-8c1e-ecc334f20d78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:30.782483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0dcd96b-8910-4985-8c1e-ecc334f20d78 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:30.838521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 904/1810] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:30 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:30.839832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dcd96b-8910-4985-8c1e-ecc334f20d78 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:30.854549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dcd96b-8910-4985-8c1e-ecc334f20d78 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:30.863714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616a12ea-90cf-4acf-80e3-b6cd7dbafe65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:30.864798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a12ea-90cf-4acf-80e3-b6cd7dbafe65 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:30.865673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a12ea-90cf-4acf-80e3-b6cd7dbafe65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:30.865882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a12ea-90cf-4acf-80e3-b6cd7dbafe65 admin admin] PATH_INFO: `/v3/projects/ebbbe8efe0bd43e4a81e5150b2d20f5c/users/2763b466fa6c435c94f53ec77f915463/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:30.876539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d5d2f6-4038-4054-8e3f-a00622d01960 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:30.894740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616a12ea-90cf-4acf-80e3-b6cd7dbafe65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebbbe8efe0bd43e4a81e5150b2d20f5c, user_id=2763b466fa6c435c94f53ec77f915463, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:30.897841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616a12ea-90cf-4acf-80e3-b6cd7dbafe65 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:30.928052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 905/1811] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:30 2022] PUT /identity/v3/projects/ebbbe8efe0bd43e4a81e5150b2d20f5c/users/2763b466fa6c435c94f53ec77f915463/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:30.933197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616a12ea-90cf-4acf-80e3-b6cd7dbafe65 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:31.085076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:31.146964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616a12ea-90cf-4acf-80e3-b6cd7dbafe65 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:31.149871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274ec6d9-9961-43e3-96fa-9d7ef57dc80b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:31.155841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ec6d9-9961-43e3-96fa-9d7ef57dc80b service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:31.156369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ec6d9-9961-43e3-96fa-9d7ef57dc80b service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:31.156641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ec6d9-9961-43e3-96fa-9d7ef57dc80b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:31.160650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa8a380-147d-46d0-89fc-727a28e007c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:31.161582 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8a380-147d-46d0-89fc-727a28e007c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:31.161582 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8a380-147d-46d0-89fc-727a28e007c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:31.167805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa8a380-147d-46d0-89fc-727a28e007c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:31.168670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa8a380-147d-46d0-89fc-727a28e007c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:31.169783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-274ec6d9-9961-43e3-96fa-9d7ef57dc80b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:31.171660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa8a380-147d-46d0-89fc-727a28e007c9 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:31.211284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 906/1812] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:30 2022] GET /identity/v3/roles => generated 1420 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:31.215102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef02975e-c6a5-4eb4-902d-2dfb641a780f None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:31.215566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef02975e-c6a5-4eb4-902d-2dfb641a780f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:31.216684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef02975e-c6a5-4eb4-902d-2dfb641a780f None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:31.219169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 907/1813] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:30:31 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:30:31.220128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef02975e-c6a5-4eb4-902d-2dfb641a780f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:31.278552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef02975e-c6a5-4eb4-902d-2dfb641a780f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:31.295749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4db45e-0a94-4cb3-ae3e-2723374a5014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:31.297407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4db45e-0a94-4cb3-ae3e-2723374a5014 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:31.298951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4db45e-0a94-4cb3-ae3e-2723374a5014 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:31.300396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4db45e-0a94-4cb3-ae3e-2723374a5014 admin admin] PATH_INFO: `/v3/projects/ebbbe8efe0bd43e4a81e5150b2d20f5c/users/2763b466fa6c435c94f53ec77f915463/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:31.306033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4db45e-0a94-4cb3-ae3e-2723374a5014 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebbbe8efe0bd43e4a81e5150b2d20f5c, user_id=2763b466fa6c435c94f53ec77f915463, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:31.312922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4db45e-0a94-4cb3-ae3e-2723374a5014 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:31.358482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 908/1814] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:31 2022] PUT /identity/v3/projects/ebbbe8efe0bd43e4a81e5150b2d20f5c/users/2763b466fa6c435c94f53ec77f915463/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:31.364628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e1f890-f609-46a1-bde3-66d889299b29 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:31.366533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e1f890-f609-46a1-bde3-66d889299b29 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:31.366892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e1f890-f609-46a1-bde3-66d889299b29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:31.442306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-65e1f890-f609-46a1-bde3-66d889299b29 None None] MFA Rules not processed for user `2763b466fa6c435c94f53ec77f915463`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:31.444386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-65e1f890-f609-46a1-bde3-66d889299b29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:31.453481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274ec6d9-9961-43e3-96fa-9d7ef57dc80b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:31.454947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274ec6d9-9961-43e3-96fa-9d7ef57dc80b service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:31.476274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 907/1815] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:30 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 920 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:30:31.478971 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b531ab0-f3cd-470c-b2eb-e31edc8f09f2 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:31.480412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b531ab0-f3cd-470c-b2eb-e31edc8f09f2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:31.480412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b531ab0-f3cd-470c-b2eb-e31edc8f09f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:31.518310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-2b531ab0-f3cd-470c-b2eb-e31edc8f09f2 None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:31.521470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2b531ab0-f3cd-470c-b2eb-e31edc8f09f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:31.742312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 908/1816] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:30:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:31.766442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 909/1817] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:31 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 403 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:31.959093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b531ab0-f3cd-470c-b2eb-e31edc8f09f2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:31.961346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2b531ab0-f3cd-470c-b2eb-e31edc8f09f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:31.989341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b531ab0-f3cd-470c-b2eb-e31edc8f09f2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:31.996655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c219d86-5e97-4f30-a932-361b5ba9f0ae None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:31.997316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c219d86-5e97-4f30-a932-361b5ba9f0ae None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:31.997677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c219d86-5e97-4f30-a932-361b5ba9f0ae None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:31.999230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 910/1818] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:30:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:30:32.005528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf6ea99-5a4b-442e-ace3-6383df89bf11 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:32.006722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf6ea99-5a4b-442e-ace3-6383df89bf11 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:32.007298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf6ea99-5a4b-442e-ace3-6383df89bf11 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:32.008374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf6ea99-5a4b-442e-ace3-6383df89bf11 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:32.040133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d6569e-4aa5-4066-911a-5af05e6920ff None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:32.040438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d6569e-4aa5-4066-911a-5af05e6920ff None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:32.042277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d6569e-4aa5-4066-911a-5af05e6920ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:32.073603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf6ea99-5a4b-442e-ace3-6383df89bf11 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:32.085623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf6ea99-5a4b-442e-ace3-6383df89bf11 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:32.103677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-95d6569e-4aa5-4066-911a-5af05e6920ff None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:32.113912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-95d6569e-4aa5-4066-911a-5af05e6920ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:32.121502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 909/1819] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:32.123248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf6ea99-5a4b-442e-ace3-6383df89bf11 service nova] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:32.165865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:32.188114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 911/1820] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:30:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:30:32.246047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf6ea99-5a4b-442e-ace3-6383df89bf11 service nova] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:32.265509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d326226-f88d-4dc9-8d16-603d8e183a1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:32.266274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d326226-f88d-4dc9-8d16-603d8e183a1b service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:32.266588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d326226-f88d-4dc9-8d16-603d8e183a1b service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:32.266859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d326226-f88d-4dc9-8d16-603d8e183a1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:32.283578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d326226-f88d-4dc9-8d16-603d8e183a1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:32.286628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d326226-f88d-4dc9-8d16-603d8e183a1b service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:32.302074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 910/1821] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:32 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:32.396710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d6569e-4aa5-4066-911a-5af05e6920ff None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:32.422388 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d6569e-4aa5-4066-911a-5af05e6920ff None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:32.439395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fca8ac-9cc6-48c2-a806-b96ec8becd2e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:32.440352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fca8ac-9cc6-48c2-a806-b96ec8becd2e service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:32.440871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fca8ac-9cc6-48c2-a806-b96ec8becd2e service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:32.441347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fca8ac-9cc6-48c2-a806-b96ec8becd2e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:32.457409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fca8ac-9cc6-48c2-a806-b96ec8becd2e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:32.459617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fca8ac-9cc6-48c2-a806-b96ec8becd2e service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:32.476549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 912/1822] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:32.774340 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d326226-f88d-4dc9-8d16-603d8e183a1b service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:32.775824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5d326226-f88d-4dc9-8d16-603d8e183a1b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:32.790094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d326226-f88d-4dc9-8d16-603d8e183a1b service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:32.805436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e782dae-b424-4cb7-9845-ef93991156ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:32.806074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e782dae-b424-4cb7-9845-ef93991156ff service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:32.806390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e782dae-b424-4cb7-9845-ef93991156ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:32.806660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e782dae-b424-4cb7-9845-ef93991156ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:32.892920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e782dae-b424-4cb7-9845-ef93991156ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:32.894670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e782dae-b424-4cb7-9845-ef93991156ff service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:32.905036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 911/1823] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:32 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:33.585479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fca8ac-9cc6-48c2-a806-b96ec8becd2e service nova] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:33.596726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fca8ac-9cc6-48c2-a806-b96ec8becd2e service nova] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:33.600910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760a8269-12db-4fba-9a42-d026537c3a7d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:33.601526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760a8269-12db-4fba-9a42-d026537c3a7d service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:33.601801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760a8269-12db-4fba-9a42-d026537c3a7d service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:33.602066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760a8269-12db-4fba-9a42-d026537c3a7d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:33.696320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760a8269-12db-4fba-9a42-d026537c3a7d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:33.698761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760a8269-12db-4fba-9a42-d026537c3a7d service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:33.711420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 913/1824] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:33 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:37.877269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e782dae-b424-4cb7-9845-ef93991156ff service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:37.952419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e782dae-b424-4cb7-9845-ef93991156ff service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:37.958017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:37.958720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:37.958999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:37.959351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] PATH_INFO: `/v3/users/02cfcbe4f42140659fb5a47230054fb6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:37.960437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=02cfcbe4f42140659fb5a47230054fb6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:37.962397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:38.013805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02cfcbe4f42140659fb5a47230054fb6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.021319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02cfcbe4f42140659fb5a47230054fb6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.022160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02cfcbe4f42140659fb5a47230054fb6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.023267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02cfcbe4f42140659fb5a47230054fb6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.030328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02cfcbe4f42140659fb5a47230054fb6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.034696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:38.049939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 912/1825] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:37 2022] DELETE /identity/v3/users/02cfcbe4f42140659fb5a47230054fb6 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:38.054758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d673769b-5e85-4f7c-a52f-44b46fd0720f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:38.055243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d673769b-5e85-4f7c-a52f-44b46fd0720f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:38.055324 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d673769b-5e85-4f7c-a52f-44b46fd0720f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:38.074250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d673769b-5e85-4f7c-a52f-44b46fd0720f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:38.075474 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d673769b-5e85-4f7c-a52f-44b46fd0720f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:38.084354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:38.133941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 914/1826] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:38.143968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:38.146131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:38.204716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2381d54-29d7-4c92-bcc8-51ea22718af5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:38.208802 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a5d869-5c44-49a8-bac7-903e7f3823fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:38.209507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a5d869-5c44-49a8-bac7-903e7f3823fa service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:38.210237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a5d869-5c44-49a8-bac7-903e7f3823fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:38.210237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a5d869-5c44-49a8-bac7-903e7f3823fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:38.219129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a5d869-5c44-49a8-bac7-903e7f3823fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:38.220544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a5d869-5c44-49a8-bac7-903e7f3823fa service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:38.230369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 913/1827] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:38.595507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d673769b-5e85-4f7c-a52f-44b46fd0720f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:38.597029 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d673769b-5e85-4f7c-a52f-44b46fd0720f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:38.615448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d673769b-5e85-4f7c-a52f-44b46fd0720f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:38.622766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:38.624027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:38.624267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:38.624529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] PATH_INFO: `/v3/projects/e6530f095c314ca18ac654c146d384f7` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:38.625649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e6530f095c314ca18ac654c146d384f7)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:38.627601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:38.686841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e6530f095c314ca18ac654c146d384f7', 'request_id': 'req-67b720db-ac78-4bbf-844d-b879a415ff06'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.694996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e6530f095c314ca18ac654c146d384f7', 'request_id': 'req-67b720db-ac78-4bbf-844d-b879a415ff06'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.711669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e6530f095c314ca18ac654c146d384f7', 'request_id': 'req-67b720db-ac78-4bbf-844d-b879a415ff06'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.729252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] The token cache is being invalidate because project e6530f095c314ca18ac654c146d384f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:38.729549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.736736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.738430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 915/1828] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:38 2022] DELETE /identity/v3/projects/e6530f095c314ca18ac654c146d384f7 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:38.743532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a5d869-5c44-49a8-bac7-903e7f3823fa service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:38.754508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a5d869-5c44-49a8-bac7-903e7f3823fa service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:38.817580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:38.818158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:38.818438 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:38.818702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] PATH_INFO: `/v3/users/2763b466fa6c435c94f53ec77f915463` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:38.819809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=2763b466fa6c435c94f53ec77f915463)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:38.821773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:38.874270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2763b466fa6c435c94f53ec77f915463'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.883230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2763b466fa6c435c94f53ec77f915463'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.884316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2763b466fa6c435c94f53ec77f915463'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.885032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2763b466fa6c435c94f53ec77f915463'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.892570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9db70ab0-5a7e-4142-85c6-97f601efb5bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2763b466fa6c435c94f53ec77f915463'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:38.911263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 914/1829] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:38 2022] DELETE /identity/v3/users/2763b466fa6c435c94f53ec77f915463 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:39.184500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:39.186043 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:39.202615 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b720db-ac78-4bbf-844d-b879a415ff06 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:39.264757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:39.265406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:39.265711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:39.265975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] PATH_INFO: `/v3/projects/ebbbe8efe0bd43e4a81e5150b2d20f5c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:39.267227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ebbbe8efe0bd43e4a81e5150b2d20f5c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:39.269208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:39.317381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:39.325892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ebbbe8efe0bd43e4a81e5150b2d20f5c', 'request_id': 'req-b0459118-7852-46c0-9452-20d473127969'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:39.334236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ebbbe8efe0bd43e4a81e5150b2d20f5c', 'request_id': 'req-b0459118-7852-46c0-9452-20d473127969'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:39.344627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ebbbe8efe0bd43e4a81e5150b2d20f5c', 'request_id': 'req-b0459118-7852-46c0-9452-20d473127969'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:39.357766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] The token cache is being invalidate because project ebbbe8efe0bd43e4a81e5150b2d20f5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:39.358189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:39.358818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b0459118-7852-46c0-9452-20d473127969 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:39.360378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 916/1830] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:39 2022] DELETE /identity/v3/projects/ebbbe8efe0bd43e4a81e5150b2d20f5c => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:39.366126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a9d2a5-a092-42a4-850a-6b1641aafb94 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:39.366459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a9d2a5-a092-42a4-850a-6b1641aafb94 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:39.366733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a9d2a5-a092-42a4-850a-6b1641aafb94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:39.381689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:39.388110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-72a9d2a5-a092-42a4-850a-6b1641aafb94 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:39.390406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-72a9d2a5-a092-42a4-850a-6b1641aafb94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:39.458804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 915/1831] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:39.471763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d0bd96-801b-4a68-8bd6-4a1aaa4b9abc None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:39.472745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d0bd96-801b-4a68-8bd6-4a1aaa4b9abc None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:39.473256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d0bd96-801b-4a68-8bd6-4a1aaa4b9abc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:39.496997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-05d0bd96-801b-4a68-8bd6-4a1aaa4b9abc None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:39.498353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-05d0bd96-801b-4a68-8bd6-4a1aaa4b9abc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:39.507695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 917/1832] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:39.512761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a9d2a5-a092-42a4-850a-6b1641aafb94 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:39.522955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a9d2a5-a092-42a4-850a-6b1641aafb94 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:39.529368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fac173b-c060-4994-b4e1-e2763c5785f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:39.530437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fac173b-c060-4994-b4e1-e2763c5785f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:39.530930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fac173b-c060-4994-b4e1-e2763c5785f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:39.531373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fac173b-c060-4994-b4e1-e2763c5785f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:39.532239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fac173b-c060-4994-b4e1-e2763c5785f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:30:39.533826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fac173b-c060-4994-b4e1-e2763c5785f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:39.536253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fac173b-c060-4994-b4e1-e2763c5785f2 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:39.552306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 916/1833] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:30:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:30:39.556572 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d0bd96-801b-4a68-8bd6-4a1aaa4b9abc None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:39.570696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d0bd96-801b-4a68-8bd6-4a1aaa4b9abc None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:39.578490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8229374-c9fe-4bf4-83f2-7834fded8070 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:39.579519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8229374-c9fe-4bf4-83f2-7834fded8070 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:39.580131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8229374-c9fe-4bf4-83f2-7834fded8070 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:39.580584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8229374-c9fe-4bf4-83f2-7834fded8070 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:39.581505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8229374-c9fe-4bf4-83f2-7834fded8070 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:39.583707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8229374-c9fe-4bf4-83f2-7834fded8070 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:39.650206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 918/1834] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:39 2022] POST /identity/v3/projects => generated 375 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:30:39.651503 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fac173b-c060-4994-b4e1-e2763c5785f2 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:39.661610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fac173b-c060-4994-b4e1-e2763c5785f2 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:39.734411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051f55f5-b1d5-4a6b-b833-6103ca62af56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:39.735719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051f55f5-b1d5-4a6b-b833-6103ca62af56 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:39.736000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051f55f5-b1d5-4a6b-b833-6103ca62af56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:39.736275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051f55f5-b1d5-4a6b-b833-6103ca62af56 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:39.736927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051f55f5-b1d5-4a6b-b833-6103ca62af56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:39.739413 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051f55f5-b1d5-4a6b-b833-6103ca62af56 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:39.794889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8229374-c9fe-4bf4-83f2-7834fded8070 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:39.795299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 917/1835] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:39 2022] POST /identity/v3/users => generated 288 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:39.813902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8229374-c9fe-4bf4-83f2-7834fded8070 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:39.823798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a715e67-6419-4285-b878-3ad291cd4751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:39.824499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a715e67-6419-4285-b878-3ad291cd4751 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:39.824779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a715e67-6419-4285-b878-3ad291cd4751 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:39.828423 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a715e67-6419-4285-b878-3ad291cd4751 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:39.829171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a715e67-6419-4285-b878-3ad291cd4751 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:39.831438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a715e67-6419-4285-b878-3ad291cd4751 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:39.849527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 919/1836] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:39 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:39.854013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051f55f5-b1d5-4a6b-b833-6103ca62af56 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:39.866885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051f55f5-b1d5-4a6b-b833-6103ca62af56 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:39.873472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5496903e-f74b-408e-aa15-13823776f377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:39.873756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5496903e-f74b-408e-aa15-13823776f377 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:39.874021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5496903e-f74b-408e-aa15-13823776f377 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:39.875946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5496903e-f74b-408e-aa15-13823776f377 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:39.876909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5496903e-f74b-408e-aa15-13823776f377 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:39.878635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5496903e-f74b-408e-aa15-13823776f377 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:39.901331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a715e67-6419-4285-b878-3ad291cd4751 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:39.902514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 918/1837] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:39 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:39.927850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a715e67-6419-4285-b878-3ad291cd4751 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:39.936755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c053796d-0e1c-4419-b982-5c83dfd118fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:39.937435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c053796d-0e1c-4419-b982-5c83dfd118fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:39.937708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c053796d-0e1c-4419-b982-5c83dfd118fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:39.937975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c053796d-0e1c-4419-b982-5c83dfd118fc admin admin] PATH_INFO: `/v3/projects/994dc860b884459e93d588d2c3625e17/users/3673c2772010473e8724590f3f91a03d/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:39.967566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c053796d-0e1c-4419-b982-5c83dfd118fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=994dc860b884459e93d588d2c3625e17, user_id=3673c2772010473e8724590f3f91a03d, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:39.972109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c053796d-0e1c-4419-b982-5c83dfd118fc admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:40.012874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 920/1838] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:39 2022] PUT /identity/v3/projects/994dc860b884459e93d588d2c3625e17/users/3673c2772010473e8724590f3f91a03d/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:40.024516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a0fac5-9b24-4051-830c-78e317959f84 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:40.026204 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a0fac5-9b24-4051-830c-78e317959f84 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:40.026725 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a0fac5-9b24-4051-830c-78e317959f84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:40.109817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c5a0fac5-9b24-4051-830c-78e317959f84 None None] MFA Rules not processed for user `3673c2772010473e8724590f3f91a03d`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:40.111364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c5a0fac5-9b24-4051-830c-78e317959f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:40.242022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 919/1839] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:40 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:40.244678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c053796d-0e1c-4419-b982-5c83dfd118fc admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:40.312419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c053796d-0e1c-4419-b982-5c83dfd118fc admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:40.317588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90992627-2c37-465e-90fb-89bfd05b2abb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:40.318502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90992627-2c37-465e-90fb-89bfd05b2abb admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:40.318787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90992627-2c37-465e-90fb-89bfd05b2abb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:40.319062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90992627-2c37-465e-90fb-89bfd05b2abb admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:40.319717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90992627-2c37-465e-90fb-89bfd05b2abb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:40.321665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90992627-2c37-465e-90fb-89bfd05b2abb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:40.354698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 921/1840] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:40 2022] POST /identity/v3/projects => generated 373 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:40.358859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a0fac5-9b24-4051-830c-78e317959f84 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:40.364642 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:40.372229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a0fac5-9b24-4051-830c-78e317959f84 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:40.434599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603e0cc1-5c07-4bbc-9525-1d25a9b8bb9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:40.435127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603e0cc1-5c07-4bbc-9525-1d25a9b8bb9a admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:40.435398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603e0cc1-5c07-4bbc-9525-1d25a9b8bb9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:40.435651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603e0cc1-5c07-4bbc-9525-1d25a9b8bb9a admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:40.436290 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603e0cc1-5c07-4bbc-9525-1d25a9b8bb9a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:40.438224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603e0cc1-5c07-4bbc-9525-1d25a9b8bb9a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:40.461511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 920/1841] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:40 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:40.465586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90992627-2c37-465e-90fb-89bfd05b2abb admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:40.476558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90992627-2c37-465e-90fb-89bfd05b2abb admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:40.482804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfae0ee-4b7b-4040-adab-1893ca50f065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:40.483479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfae0ee-4b7b-4040-adab-1893ca50f065 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:40.483752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfae0ee-4b7b-4040-adab-1893ca50f065 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:40.484017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfae0ee-4b7b-4040-adab-1893ca50f065 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:40.484788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfae0ee-4b7b-4040-adab-1893ca50f065 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:40.486566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfae0ee-4b7b-4040-adab-1893ca50f065 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:40.498202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 922/1842] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:40 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:40.502482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603e0cc1-5c07-4bbc-9525-1d25a9b8bb9a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:40.513090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603e0cc1-5c07-4bbc-9525-1d25a9b8bb9a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:40.518240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40c29d9-e9d2-4289-b3f7-91950d80bb07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:40.518932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40c29d9-e9d2-4289-b3f7-91950d80bb07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:40.519207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40c29d9-e9d2-4289-b3f7-91950d80bb07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:40.519473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40c29d9-e9d2-4289-b3f7-91950d80bb07 admin admin] PATH_INFO: `/v3/projects/96e5aeee1a1e48988b695c18c90a9421/users/21c155f110d1499ea86c1c745edd569a/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:40.545776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40c29d9-e9d2-4289-b3f7-91950d80bb07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=96e5aeee1a1e48988b695c18c90a9421, user_id=21c155f110d1499ea86c1c745edd569a, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:40.548525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40c29d9-e9d2-4289-b3f7-91950d80bb07 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:40.564658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 921/1843] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:40 2022] PUT /identity/v3/projects/96e5aeee1a1e48988b695c18c90a9421/users/21c155f110d1499ea86c1c745edd569a/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:40.568748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfae0ee-4b7b-4040-adab-1893ca50f065 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:40.579923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfae0ee-4b7b-4040-adab-1893ca50f065 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:40.629365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:40.654258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366b3d2f-b789-4726-850d-be10436fa23f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:40.655093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366b3d2f-b789-4726-850d-be10436fa23f admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:40.655461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366b3d2f-b789-4726-850d-be10436fa23f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:40.655955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366b3d2f-b789-4726-850d-be10436fa23f admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:40.656897 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366b3d2f-b789-4726-850d-be10436fa23f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:40.658821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366b3d2f-b789-4726-850d-be10436fa23f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:40.670192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 923/1844] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:40 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:30:40.674468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40c29d9-e9d2-4289-b3f7-91950d80bb07 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:40.685404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40c29d9-e9d2-4289-b3f7-91950d80bb07 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:40.690602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58934a3-25cc-4ad4-8adc-1b62c6447aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:40.691356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58934a3-25cc-4ad4-8adc-1b62c6447aa8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:40.691634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58934a3-25cc-4ad4-8adc-1b62c6447aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:40.691909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58934a3-25cc-4ad4-8adc-1b62c6447aa8 admin admin] PATH_INFO: `/v3/projects/96e5aeee1a1e48988b695c18c90a9421/users/21c155f110d1499ea86c1c745edd569a/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:40.693842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58934a3-25cc-4ad4-8adc-1b62c6447aa8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=96e5aeee1a1e48988b695c18c90a9421, user_id=21c155f110d1499ea86c1c745edd569a, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:40.696545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58934a3-25cc-4ad4-8adc-1b62c6447aa8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:40.708300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 922/1845] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:40 2022] PUT /identity/v3/projects/96e5aeee1a1e48988b695c18c90a9421/users/21c155f110d1499ea86c1c745edd569a/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:30:40.719022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b52dee8-83e7-4758-9bc8-2b78d92806d5 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:40.719626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b52dee8-83e7-4758-9bc8-2b78d92806d5 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:40.720071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b52dee8-83e7-4758-9bc8-2b78d92806d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:40.780586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2b52dee8-83e7-4758-9bc8-2b78d92806d5 None None] MFA Rules not processed for user `21c155f110d1499ea86c1c745edd569a`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:40.782271 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2b52dee8-83e7-4758-9bc8-2b78d92806d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:40.866135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 924/1846] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:40 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:30:40.879269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58934a3-25cc-4ad4-8adc-1b62c6447aa8 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:40.880872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d58934a3-25cc-4ad4-8adc-1b62c6447aa8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:40.949088 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58934a3-25cc-4ad4-8adc-1b62c6447aa8 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:40.953745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8edd15c-f73c-47a6-a02b-0607b9f9b8fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:40.954515 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8edd15c-f73c-47a6-a02b-0607b9f9b8fd service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:40.954963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8edd15c-f73c-47a6-a02b-0607b9f9b8fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:40.955345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8edd15c-f73c-47a6-a02b-0607b9f9b8fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:41.021516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8edd15c-f73c-47a6-a02b-0607b9f9b8fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:41.022991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8edd15c-f73c-47a6-a02b-0607b9f9b8fd service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:41.032585 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 923/1847] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:40 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:41.978883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b52dee8-83e7-4758-9bc8-2b78d92806d5 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:41.980691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2b52dee8-83e7-4758-9bc8-2b78d92806d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:42.045685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b52dee8-83e7-4758-9bc8-2b78d92806d5 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:42.050683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:42.051299 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:42.051568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:42.051822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] PATH_INFO: `/v3/users/1899d9dbb96e43f0b1fa5f0cac66352c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:42.053001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1899d9dbb96e43f0b1fa5f0cac66352c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:42.054947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:42.114339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1899d9dbb96e43f0b1fa5f0cac66352c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:42.115017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1899d9dbb96e43f0b1fa5f0cac66352c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:42.122428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1899d9dbb96e43f0b1fa5f0cac66352c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:42.123066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1899d9dbb96e43f0b1fa5f0cac66352c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:42.132108 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1899d9dbb96e43f0b1fa5f0cac66352c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:42.133224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:42.154929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 925/1848] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:41 2022] DELETE /identity/v3/users/1899d9dbb96e43f0b1fa5f0cac66352c => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:42.159560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe5131d-d2aa-4944-b439-f8b0b52934ac None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:42.160110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe5131d-d2aa-4944-b439-f8b0b52934ac None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:42.160110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe5131d-d2aa-4944-b439-f8b0b52934ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:42.192761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-bbe5131d-d2aa-4944-b439-f8b0b52934ac None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:42.193967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-bbe5131d-d2aa-4944-b439-f8b0b52934ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:42.198824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:42.261851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 924/1849] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:42.271822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:42.273252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:42.333605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf921c8d-d083-4d3a-8dea-1afe1a78bbd1 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:42.337489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740b2ab-2d93-4f09-b866-b38fa55527dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:42.338181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740b2ab-2d93-4f09-b866-b38fa55527dd service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:42.338533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740b2ab-2d93-4f09-b866-b38fa55527dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:42.338732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740b2ab-2d93-4f09-b866-b38fa55527dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:42.348647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b740b2ab-2d93-4f09-b866-b38fa55527dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:42.350260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b740b2ab-2d93-4f09-b866-b38fa55527dd service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:42.359643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 926/1850] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:42.835570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe5131d-d2aa-4944-b439-f8b0b52934ac None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:42.837256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-bbe5131d-d2aa-4944-b439-f8b0b52934ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:42.851822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe5131d-d2aa-4944-b439-f8b0b52934ac None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:42.867221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:42.868564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:42.868564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:42.868778 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] PATH_INFO: `/v3/projects/146a77f3ce4c4463ae8e9af9731710c7` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:42.870018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=146a77f3ce4c4463ae8e9af9731710c7)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:42.872063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:42.933632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '146a77f3ce4c4463ae8e9af9731710c7', 'request_id': 'req-2eee860f-b5e0-4e43-b2ff-63d6680562e0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:42.941295 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '146a77f3ce4c4463ae8e9af9731710c7', 'request_id': 'req-2eee860f-b5e0-4e43-b2ff-63d6680562e0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:42.949943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '146a77f3ce4c4463ae8e9af9731710c7', 'request_id': 'req-2eee860f-b5e0-4e43-b2ff-63d6680562e0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:42.967294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] The token cache is being invalidate because project 146a77f3ce4c4463ae8e9af9731710c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:42.967918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:42.969134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:42.971518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 925/1851] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:42 2022] DELETE /identity/v3/projects/146a77f3ce4c4463ae8e9af9731710c7 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:42.975902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740b2ab-2d93-4f09-b866-b38fa55527dd service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:42.986756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740b2ab-2d93-4f09-b866-b38fa55527dd service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:43.069845 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:43.070954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:43.071302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:43.071791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] PATH_INFO: `/v3/users/528cfa4a906347e79698f3dcb89ee645` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:43.073071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] RBAC: Authorizing `identity:delete_user(user_id=528cfa4a906347e79698f3dcb89ee645)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:43.075623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:43.132614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '528cfa4a906347e79698f3dcb89ee645'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.133657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '528cfa4a906347e79698f3dcb89ee645'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.142660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '528cfa4a906347e79698f3dcb89ee645'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.143426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '528cfa4a906347e79698f3dcb89ee645'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.151931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3446b899-4825-4959-adda-6ba9ef85970c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '528cfa4a906347e79698f3dcb89ee645'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.170048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 927/1852] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:42 2022] DELETE /identity/v3/users/528cfa4a906347e79698f3dcb89ee645 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:43.455480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:43.457014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:43.471577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eee860f-b5e0-4e43-b2ff-63d6680562e0 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:43.533070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:43.533716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:43.534018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:43.534311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] PATH_INFO: `/v3/projects/ea0448df78df427dbfac27521be7a626` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:43.535555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea0448df78df427dbfac27521be7a626)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:43.537506 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:43.580399 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:43.592235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea0448df78df427dbfac27521be7a626', 'request_id': 'req-12b2da50-22dc-40e5-93d3-6cf2b155da73'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.598303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea0448df78df427dbfac27521be7a626', 'request_id': 'req-12b2da50-22dc-40e5-93d3-6cf2b155da73'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.607834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea0448df78df427dbfac27521be7a626', 'request_id': 'req-12b2da50-22dc-40e5-93d3-6cf2b155da73'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.622174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] The token cache is being invalidate because project ea0448df78df427dbfac27521be7a626 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:43.622387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.623029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b2da50-22dc-40e5-93d3-6cf2b155da73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.627955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 926/1853] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:43 2022] DELETE /identity/v3/projects/ea0448df78df427dbfac27521be7a626 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:43.630671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5e47ac-e7a4-4b59-bfcd-6ea90e14401c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:43.631031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5e47ac-e7a4-4b59-bfcd-6ea90e14401c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:43.631370 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5e47ac-e7a4-4b59-bfcd-6ea90e14401c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:43.640178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:43.657104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-bb5e47ac-e7a4-4b59-bfcd-6ea90e14401c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:43.658415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-bb5e47ac-e7a4-4b59-bfcd-6ea90e14401c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:43.721378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 928/1854] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:43.729800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81562c45-6953-409a-ac1d-08ee9f7e83b0 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:43.730204 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81562c45-6953-409a-ac1d-08ee9f7e83b0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:43.730481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81562c45-6953-409a-ac1d-08ee9f7e83b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:43.754060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-81562c45-6953-409a-ac1d-08ee9f7e83b0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:43.755445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-81562c45-6953-409a-ac1d-08ee9f7e83b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:43.765489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 927/1855] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:43.770565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5e47ac-e7a4-4b59-bfcd-6ea90e14401c None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:43.780341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5e47ac-e7a4-4b59-bfcd-6ea90e14401c None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:43.785159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4295e29-784c-4ad0-a377-d489779675ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:43.786740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4295e29-784c-4ad0-a377-d489779675ba admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:43.787027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4295e29-784c-4ad0-a377-d489779675ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:43.787299 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4295e29-784c-4ad0-a377-d489779675ba admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:43.787964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4295e29-784c-4ad0-a377-d489779675ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:30:43.788269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4295e29-784c-4ad0-a377-d489779675ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:43.803025 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4295e29-784c-4ad0-a377-d489779675ba admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:43.816885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81562c45-6953-409a-ac1d-08ee9f7e83b0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:43.818320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-81562c45-6953-409a-ac1d-08ee9f7e83b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:43.825907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 929/1856] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:30:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:30:43.830153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4295e29-784c-4ad0-a377-d489779675ba admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:43.832615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81562c45-6953-409a-ac1d-08ee9f7e83b0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:43.838440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:43.839719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:43.839921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:43.840200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] PATH_INFO: `/v3/users/3673c2772010473e8724590f3f91a03d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:43.841877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3673c2772010473e8724590f3f91a03d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:43.844354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4295e29-784c-4ad0-a377-d489779675ba admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:43.844490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:43.850807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58d12db-2ae4-46bb-baf2-cc9f378d97ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:43.851590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58d12db-2ae4-46bb-baf2-cc9f378d97ec admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:43.852394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58d12db-2ae4-46bb-baf2-cc9f378d97ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:43.853750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58d12db-2ae4-46bb-baf2-cc9f378d97ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:43.854645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58d12db-2ae4-46bb-baf2-cc9f378d97ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:43.856738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58d12db-2ae4-46bb-baf2-cc9f378d97ec admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:43.904729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 930/1857] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:43 2022] POST /identity/v3/projects => generated 387 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:43.905653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58d12db-2ae4-46bb-baf2-cc9f378d97ec admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:43.907752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3673c2772010473e8724590f3f91a03d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.919399 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3673c2772010473e8724590f3f91a03d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.921891 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3673c2772010473e8724590f3f91a03d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.923273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3673c2772010473e8724590f3f91a03d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.934544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f704ca81-59d1-4516-91b3-d3a912cfa3e4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3673c2772010473e8724590f3f91a03d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:43.962578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 928/1858] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:43 2022] DELETE /identity/v3/users/3673c2772010473e8724590f3f91a03d => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:43.965127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e43f01f-70f1-44c3-b84e-f7e4a098549a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:43.966151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e43f01f-70f1-44c3-b84e-f7e4a098549a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:43.966578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e43f01f-70f1-44c3-b84e-f7e4a098549a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:43.992337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58d12db-2ae4-46bb-baf2-cc9f378d97ec admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:44.020846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-3e43f01f-70f1-44c3-b84e-f7e4a098549a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:44.023062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3e43f01f-70f1-44c3-b84e-f7e4a098549a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:44.082356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ac8d75-8209-477e-bdb6-b28e6c85acd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:44.082356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ac8d75-8209-477e-bdb6-b28e6c85acd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:44.082356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ac8d75-8209-477e-bdb6-b28e6c85acd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:44.082356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ac8d75-8209-477e-bdb6-b28e6c85acd7 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:44.083015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ac8d75-8209-477e-bdb6-b28e6c85acd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:44.084790 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ac8d75-8209-477e-bdb6-b28e6c85acd7 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:44.096913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 929/1859] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:44.124218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e43f01f-70f1-44c3-b84e-f7e4a098549a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:44.124218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3e43f01f-70f1-44c3-b84e-f7e4a098549a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:44.124405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 931/1860] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:43 2022] POST /identity/v3/users => generated 294 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:44.124405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ac8d75-8209-477e-bdb6-b28e6c85acd7 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:44.124405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e8ac8d75-8209-477e-bdb6-b28e6c85acd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:44.126966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:44.130595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ac8d75-8209-477e-bdb6-b28e6c85acd7 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:44.138490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17c5757-371c-4102-b55a-c4162593f362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:44.140109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c5757-371c-4102-b55a-c4162593f362 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:44.140814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c5757-371c-4102-b55a-c4162593f362 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:44.141443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c5757-371c-4102-b55a-c4162593f362 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:44.142181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17c5757-371c-4102-b55a-c4162593f362 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:44.143868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17c5757-371c-4102-b55a-c4162593f362 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:44.147979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:44.156169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 932/1861] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:44 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:44.160387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17c5757-371c-4102-b55a-c4162593f362 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:44.172207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17c5757-371c-4102-b55a-c4162593f362 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:44.177256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4a06ea-ec23-4105-b5fb-af2955341c9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:44.177920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4a06ea-ec23-4105-b5fb-af2955341c9d admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:44.178226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4a06ea-ec23-4105-b5fb-af2955341c9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:44.178496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4a06ea-ec23-4105-b5fb-af2955341c9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:44.179132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4a06ea-ec23-4105-b5fb-af2955341c9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:44.181002 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4a06ea-ec23-4105-b5fb-af2955341c9d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:44.181250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e43f01f-70f1-44c3-b84e-f7e4a098549a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:44.185988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1881297-0b9e-48be-b4fc-a3598d64155e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:44.186929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1881297-0b9e-48be-b4fc-a3598d64155e service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:44.187876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1881297-0b9e-48be-b4fc-a3598d64155e service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:44.188349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1881297-0b9e-48be-b4fc-a3598d64155e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:44.194219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 933/1862] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:44 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:44.198786 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4a06ea-ec23-4105-b5fb-af2955341c9d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:44.203059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1881297-0b9e-48be-b4fc-a3598d64155e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:44.204760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1881297-0b9e-48be-b4fc-a3598d64155e service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:44.223222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4a06ea-ec23-4105-b5fb-af2955341c9d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:44.230393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 930/1863] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:44.232605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ed4634-61de-4ba1-a833-3d2f08f4b677 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:44.233549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ed4634-61de-4ba1-a833-3d2f08f4b677 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:44.233902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ed4634-61de-4ba1-a833-3d2f08f4b677 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:44.234282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ed4634-61de-4ba1-a833-3d2f08f4b677 admin admin] PATH_INFO: `/v3/projects/f0af44bbcb674bcfbb8167f3e02e8931/users/be8c87b464534aa9837eb9c1afcaaadb/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:44.278364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ed4634-61de-4ba1-a833-3d2f08f4b677 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0af44bbcb674bcfbb8167f3e02e8931, user_id=be8c87b464534aa9837eb9c1afcaaadb, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:44.281514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ed4634-61de-4ba1-a833-3d2f08f4b677 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:44.322193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 934/1864] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:44 2022] PUT /identity/v3/projects/f0af44bbcb674bcfbb8167f3e02e8931/users/be8c87b464534aa9837eb9c1afcaaadb/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:30:44.330291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4003fb-920c-4011-8b9b-6139d778e4e5 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:44.330573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4003fb-920c-4011-8b9b-6139d778e4e5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:44.330908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4003fb-920c-4011-8b9b-6139d778e4e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:44.378383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-dd4003fb-920c-4011-8b9b-6139d778e4e5 None None] MFA Rules not processed for user `be8c87b464534aa9837eb9c1afcaaadb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:44.380468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-dd4003fb-920c-4011-8b9b-6139d778e4e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:44.504944 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 931/1865] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:44 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:44.509177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ed4634-61de-4ba1-a833-3d2f08f4b677 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:44.584661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ed4634-61de-4ba1-a833-3d2f08f4b677 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:44.590715 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59079650-5729-4d5e-a23a-c2e516a94064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:44.591492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59079650-5729-4d5e-a23a-c2e516a94064 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:44.591823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59079650-5729-4d5e-a23a-c2e516a94064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:44.592273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59079650-5729-4d5e-a23a-c2e516a94064 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:44.593081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59079650-5729-4d5e-a23a-c2e516a94064 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:44.595337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59079650-5729-4d5e-a23a-c2e516a94064 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:44.636257 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 935/1866] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:44 2022] POST /identity/v3/projects => generated 387 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:44.640624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4003fb-920c-4011-8b9b-6139d778e4e5 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:44.658358 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4003fb-920c-4011-8b9b-6139d778e4e5 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:44.736127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed84fa56-c4bc-4131-bc13-7ff2e34422fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:44.736809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed84fa56-c4bc-4131-bc13-7ff2e34422fd admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:44.737208 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed84fa56-c4bc-4131-bc13-7ff2e34422fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:44.737482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed84fa56-c4bc-4131-bc13-7ff2e34422fd admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:44.738346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed84fa56-c4bc-4131-bc13-7ff2e34422fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:44.740622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed84fa56-c4bc-4131-bc13-7ff2e34422fd admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:44.781619 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 932/1867] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:44 2022] POST /identity/v3/users => generated 300 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:44.785822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59079650-5729-4d5e-a23a-c2e516a94064 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:44.820760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59079650-5729-4d5e-a23a-c2e516a94064 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:44.828018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98817a78-cd2a-4daf-a772-f4c52ff5a955 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:44.829296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98817a78-cd2a-4daf-a772-f4c52ff5a955 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:44.829641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98817a78-cd2a-4daf-a772-f4c52ff5a955 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:44.829932 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98817a78-cd2a-4daf-a772-f4c52ff5a955 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:44.830618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98817a78-cd2a-4daf-a772-f4c52ff5a955 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:44.832558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98817a78-cd2a-4daf-a772-f4c52ff5a955 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:44.855453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 936/1868] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:44 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:30:44.856107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed84fa56-c4bc-4131-bc13-7ff2e34422fd admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:44.868556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed84fa56-c4bc-4131-bc13-7ff2e34422fd admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:44.875141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4946f7-3e8d-4143-b0d1-6ed883d5ca50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:44.875924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4946f7-3e8d-4143-b0d1-6ed883d5ca50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:44.876274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4946f7-3e8d-4143-b0d1-6ed883d5ca50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:44.876764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4946f7-3e8d-4143-b0d1-6ed883d5ca50 admin admin] PATH_INFO: `/v3/projects/8e1b08d66c6e4b2cb850a3eb972586bb/users/ad90ee2469784d758b0ac8884afaf401/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:44.900108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4946f7-3e8d-4143-b0d1-6ed883d5ca50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e1b08d66c6e4b2cb850a3eb972586bb, user_id=ad90ee2469784d758b0ac8884afaf401, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:44.903151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4946f7-3e8d-4143-b0d1-6ed883d5ca50 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:44.924351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 933/1869] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:44 2022] PUT /identity/v3/projects/8e1b08d66c6e4b2cb850a3eb972586bb/users/ad90ee2469784d758b0ac8884afaf401/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:30:44.925874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98817a78-cd2a-4daf-a772-f4c52ff5a955 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:44.937194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98817a78-cd2a-4daf-a772-f4c52ff5a955 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:45.006608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36fb598-eb5e-4872-9b0a-fb244b66734a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:45.007379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36fb598-eb5e-4872-9b0a-fb244b66734a admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:45.008276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36fb598-eb5e-4872-9b0a-fb244b66734a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:45.008276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36fb598-eb5e-4872-9b0a-fb244b66734a admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:45.009083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36fb598-eb5e-4872-9b0a-fb244b66734a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:45.011146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36fb598-eb5e-4872-9b0a-fb244b66734a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:45.028801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 937/1870] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:44 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:45.028997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4946f7-3e8d-4143-b0d1-6ed883d5ca50 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:45.040868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4946f7-3e8d-4143-b0d1-6ed883d5ca50 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:45.046268 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc084b9e-a9ec-4f7a-843e-aac3902a860d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:45.047038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc084b9e-a9ec-4f7a-843e-aac3902a860d admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:45.047432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc084b9e-a9ec-4f7a-843e-aac3902a860d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:45.047787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc084b9e-a9ec-4f7a-843e-aac3902a860d admin admin] PATH_INFO: `/v3/projects/8e1b08d66c6e4b2cb850a3eb972586bb/users/ad90ee2469784d758b0ac8884afaf401/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:45.049739 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc084b9e-a9ec-4f7a-843e-aac3902a860d admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e1b08d66c6e4b2cb850a3eb972586bb, user_id=ad90ee2469784d758b0ac8884afaf401, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:45.052860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc084b9e-a9ec-4f7a-843e-aac3902a860d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:45.070204 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 934/1871] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:45 2022] PUT /identity/v3/projects/8e1b08d66c6e4b2cb850a3eb972586bb/users/ad90ee2469784d758b0ac8884afaf401/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:45.079144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6d1826-1866-4f70-9a5c-9498b2ba508c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:45.079425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6d1826-1866-4f70-9a5c-9498b2ba508c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:45.079694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6d1826-1866-4f70-9a5c-9498b2ba508c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:45.162144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8c6d1826-1866-4f70-9a5c-9498b2ba508c None None] MFA Rules not processed for user `ad90ee2469784d758b0ac8884afaf401`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:45.163551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8c6d1826-1866-4f70-9a5c-9498b2ba508c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:45.204805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:45.223970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc084b9e-a9ec-4f7a-843e-aac3902a860d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:45.232503 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-cc084b9e-a9ec-4f7a-843e-aac3902a860d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:45.267571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:45.337546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 938/1872] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:45 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:45.344930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc084b9e-a9ec-4f7a-843e-aac3902a860d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:45.345591 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6d1826-1866-4f70-9a5c-9498b2ba508c None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:45.347442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8c6d1826-1866-4f70-9a5c-9498b2ba508c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:45.350210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:45.351634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:45.351634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:45.351634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] PATH_INFO: `/v3/projects/994dc860b884459e93d588d2c3625e17` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:45.352983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] RBAC: Authorizing `identity:delete_project(project_id=994dc860b884459e93d588d2c3625e17)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:45.355207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:45.428323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6d1826-1866-4f70-9a5c-9498b2ba508c None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:45.453864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '994dc860b884459e93d588d2c3625e17', 'request_id': 'req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.459940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '994dc860b884459e93d588d2c3625e17', 'request_id': 'req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.469439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '994dc860b884459e93d588d2c3625e17', 'request_id': 'req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.483764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] The token cache is being invalidate because project 994dc860b884459e93d588d2c3625e17 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:45.484234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.486561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.488395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 935/1873] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:45 2022] DELETE /identity/v3/projects/994dc860b884459e93d588d2c3625e17 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:45.490186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90d543-397c-4363-9943-6c2503ab46da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:45.491063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90d543-397c-4363-9943-6c2503ab46da service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:45.491500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90d543-397c-4363-9943-6c2503ab46da service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:45.492567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:45.492896 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90d543-397c-4363-9943-6c2503ab46da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:45.493963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:45.497173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fe90d543-397c-4363-9943-6c2503ab46da service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:45.569991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3157a761-8bc3-4b43-91ff-c0b6d8ef7990 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:45.574761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe90d543-397c-4363-9943-6c2503ab46da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:45.575558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:45.576255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe90d543-397c-4363-9943-6c2503ab46da service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:45.576924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:45.577159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:45.577434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] PATH_INFO: `/v3/users/21c155f110d1499ea86c1c745edd569a` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:45.578730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21c155f110d1499ea86c1c745edd569a)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:45.580651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:45.586919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 939/1874] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:45 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:30:45.637610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:45.639967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21c155f110d1499ea86c1c745edd569a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.656558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21c155f110d1499ea86c1c745edd569a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.657192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21c155f110d1499ea86c1c745edd569a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.658128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21c155f110d1499ea86c1c745edd569a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.669745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0b7bee7f-fd46-44dc-ad48-865cc761fcb7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21c155f110d1499ea86c1c745edd569a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.689373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 936/1875] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:45 2022] DELETE /identity/v3/users/21c155f110d1499ea86c1c745edd569a => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:45.737075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90d543-397c-4363-9943-6c2503ab46da service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:45.738758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fe90d543-397c-4363-9943-6c2503ab46da service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:45.754699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90d543-397c-4363-9943-6c2503ab46da service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:45.774118 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:45.862812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:45.863726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:45.864100 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:45.864537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] PATH_INFO: `/v3/users/be8c87b464534aa9837eb9c1afcaaadb` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:45.866402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be8c87b464534aa9837eb9c1afcaaadb)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:45.868803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:45.944436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be8c87b464534aa9837eb9c1afcaaadb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.945276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be8c87b464534aa9837eb9c1afcaaadb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.953350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be8c87b464534aa9837eb9c1afcaaadb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.954303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be8c87b464534aa9837eb9c1afcaaadb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:45.968637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be8c87b464534aa9837eb9c1afcaaadb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.003840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 940/1876] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:45 2022] DELETE /identity/v3/users/be8c87b464534aa9837eb9c1afcaaadb => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:46.008116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7976a29-1f67-4bb8-9c8e-f11237485eec None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.008563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7976a29-1f67-4bb8-9c8e-f11237485eec None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.008948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7976a29-1f67-4bb8-9c8e-f11237485eec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.042520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c7976a29-1f67-4bb8-9c8e-f11237485eec None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:46.043908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c7976a29-1f67-4bb8-9c8e-f11237485eec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:46.057381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:46.064046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:46.158313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 937/1877] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:46.167282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7976a29-1f67-4bb8-9c8e-f11237485eec None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:46.169141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c7976a29-1f67-4bb8-9c8e-f11237485eec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:46.178000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de95a0f7-dede-4386-8cc5-b18d134b5ef8 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:46.184040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:46.184825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.185109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.185387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] PATH_INFO: `/v3/projects/96e5aeee1a1e48988b695c18c90a9421` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.187131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] RBAC: Authorizing `identity:delete_project(project_id=96e5aeee1a1e48988b695c18c90a9421)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:46.189331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:46.234890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:46.243688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '96e5aeee1a1e48988b695c18c90a9421', 'request_id': 'req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.243982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7976a29-1f67-4bb8-9c8e-f11237485eec None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:46.251792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '96e5aeee1a1e48988b695c18c90a9421', 'request_id': 'req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.259470 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '96e5aeee1a1e48988b695c18c90a9421', 'request_id': 'req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.271266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] The token cache is being invalidate because project 96e5aeee1a1e48988b695c18c90a9421 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:46.271555 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.272195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7cb85c23-d6dc-4858-9ea5-c25aeb542d07 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.273993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 941/1878] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:46 2022] DELETE /identity/v3/projects/96e5aeee1a1e48988b695c18c90a9421 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:46.280043 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aa1d09-1cf5-465d-8b5f-060b9899b4d4 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.280329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aa1d09-1cf5-465d-8b5f-060b9899b4d4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.280602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aa1d09-1cf5-465d-8b5f-060b9899b4d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.296073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fba0ca0-861e-438e-a5e1-100e7434921d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:46.296701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fba0ca0-861e-438e-a5e1-100e7434921d service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.296980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fba0ca0-861e-438e-a5e1-100e7434921d service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.297253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fba0ca0-861e-438e-a5e1-100e7434921d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.299030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9fba0ca0-861e-438e-a5e1-100e7434921d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:46.301172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-b5aa1d09-1cf5-465d-8b5f-060b9899b4d4 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:46.302442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b5aa1d09-1cf5-465d-8b5f-060b9899b4d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:46.332433 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:46.364155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 942/1879] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:30:46.369371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fba0ca0-861e-438e-a5e1-100e7434921d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:46.371102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fba0ca0-861e-438e-a5e1-100e7434921d service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:46.371901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e58dc8-1d96-4cac-b39b-34ed0a748224 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.372188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e58dc8-1d96-4cac-b39b-34ed0a748224 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.372461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e58dc8-1d96-4cac-b39b-34ed0a748224 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.381297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 938/1880] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:30:46.392384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-66e58dc8-1d96-4cac-b39b-34ed0a748224 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:46.393600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-66e58dc8-1d96-4cac-b39b-34ed0a748224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:46.402343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 943/1881] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:30:46.406336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fba0ca0-861e-438e-a5e1-100e7434921d service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:46.417375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fba0ca0-861e-438e-a5e1-100e7434921d service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:46.423394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5645e63f-ddeb-452e-b184-9e1a47c399f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:46.424058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645e63f-ddeb-452e-b184-9e1a47c399f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.424339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645e63f-ddeb-452e-b184-9e1a47c399f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.424683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645e63f-ddeb-452e-b184-9e1a47c399f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.425439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5645e63f-ddeb-452e-b184-9e1a47c399f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:30:46.425889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5645e63f-ddeb-452e-b184-9e1a47c399f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:46.428115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5645e63f-ddeb-452e-b184-9e1a47c399f4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:46.441420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 939/1882] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:30:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:30:46.446277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e58dc8-1d96-4cac-b39b-34ed0a748224 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:46.456879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e58dc8-1d96-4cac-b39b-34ed0a748224 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:46.461889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa7155c-b44f-4014-b472-e87da87209c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:46.462540 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa7155c-b44f-4014-b472-e87da87209c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.462996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa7155c-b44f-4014-b472-e87da87209c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.463509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa7155c-b44f-4014-b472-e87da87209c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.464182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa7155c-b44f-4014-b472-e87da87209c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:46.466276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa7155c-b44f-4014-b472-e87da87209c1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:46.509562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 944/1883] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:46 2022] POST /identity/v3/projects => generated 373 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:30:46.513869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5645e63f-ddeb-452e-b184-9e1a47c399f4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:46.526446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5645e63f-ddeb-452e-b184-9e1a47c399f4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:46.760459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba81613b-b9be-4517-af2d-22565fb61cba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:46.761217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba81613b-b9be-4517-af2d-22565fb61cba admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.761574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba81613b-b9be-4517-af2d-22565fb61cba admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.761883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba81613b-b9be-4517-af2d-22565fb61cba admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.762793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba81613b-b9be-4517-af2d-22565fb61cba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:46.765774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba81613b-b9be-4517-af2d-22565fb61cba admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:46.781233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa7155c-b44f-4014-b472-e87da87209c1 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:46.783031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4aa7155c-b44f-4014-b472-e87da87209c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:46.803457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 940/1884] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:46 2022] POST /identity/v3/users => generated 287 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:46.806344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba81613b-b9be-4517-af2d-22565fb61cba admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:46.811712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa7155c-b44f-4014-b472-e87da87209c1 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:46.817546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba81613b-b9be-4517-af2d-22565fb61cba admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:46.818321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:46.818997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.819372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.819610 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] PATH_INFO: `/v3/projects/f0af44bbcb674bcfbb8167f3e02e8931` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.820787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0af44bbcb674bcfbb8167f3e02e8931)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:46.822878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:46.823074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40f2977-8e43-4b51-a761-ccf1b65ecab7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:46.823931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40f2977-8e43-4b51-a761-ccf1b65ecab7 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.824239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40f2977-8e43-4b51-a761-ccf1b65ecab7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.824713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40f2977-8e43-4b51-a761-ccf1b65ecab7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.825471 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40f2977-8e43-4b51-a761-ccf1b65ecab7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:46.827561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40f2977-8e43-4b51-a761-ccf1b65ecab7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:46.844435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 941/1885] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:46 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:30:46.845646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40f2977-8e43-4b51-a761-ccf1b65ecab7 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:46.860190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40f2977-8e43-4b51-a761-ccf1b65ecab7 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:46.874304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9a047a-7420-43fd-be24-03307030a040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:46.874993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a047a-7420-43fd-be24-03307030a040 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:46.875386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a047a-7420-43fd-be24-03307030a040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:46.875658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9a047a-7420-43fd-be24-03307030a040 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:46.876441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9a047a-7420-43fd-be24-03307030a040 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:46.878072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9a047a-7420-43fd-be24-03307030a040 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:46.889137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 942/1886] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:46 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:30:46.895318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9a047a-7420-43fd-be24-03307030a040 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:46.907868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9a047a-7420-43fd-be24-03307030a040 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:46.916904 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0af44bbcb674bcfbb8167f3e02e8931', 'request_id': 'req-1f2fc32c-d0e5-48c0-b720-fab07434a519'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.943971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0af44bbcb674bcfbb8167f3e02e8931', 'request_id': 'req-1f2fc32c-d0e5-48c0-b720-fab07434a519'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.948645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0af44bbcb674bcfbb8167f3e02e8931', 'request_id': 'req-1f2fc32c-d0e5-48c0-b720-fab07434a519'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.968377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] The token cache is being invalidate because project f0af44bbcb674bcfbb8167f3e02e8931 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:46.968670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.969403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:46.973139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 945/1887] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:46 2022] DELETE /identity/v3/projects/f0af44bbcb674bcfbb8167f3e02e8931 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:30:46.976965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:46.981993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:47.012792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc604276-7537-4048-a59f-f795001306e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:47.013495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc604276-7537-4048-a59f-f795001306e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:47.013805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc604276-7537-4048-a59f-f795001306e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:47.014035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc604276-7537-4048-a59f-f795001306e2 admin admin] PATH_INFO: `/v3/projects/5b820ea465db4802aaa22dbaa4e7397a/users/5ae72d51e9004670a50362d1206fab21/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:47.026562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:47.032135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc604276-7537-4048-a59f-f795001306e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b820ea465db4802aaa22dbaa4e7397a, user_id=5ae72d51e9004670a50362d1206fab21, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:47.037594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc604276-7537-4048-a59f-f795001306e2 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:47.061954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 943/1888] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:46 2022] PUT /identity/v3/projects/5b820ea465db4802aaa22dbaa4e7397a/users/5ae72d51e9004670a50362d1206fab21/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:47.071890 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8051fe9-1ece-482b-8169-5ba77596981c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:47.072369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8051fe9-1ece-482b-8169-5ba77596981c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:47.072851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8051fe9-1ece-482b-8169-5ba77596981c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:47.075805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2fc32c-d0e5-48c0-b720-fab07434a519 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:47.088172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:47.094838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e8051fe9-1ece-482b-8169-5ba77596981c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:47.097477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e8051fe9-1ece-482b-8169-5ba77596981c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:47.190171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:47.191988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:47.192270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:47.192544 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] PATH_INFO: `/v3/users/ad90ee2469784d758b0ac8884afaf401` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:47.195975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad90ee2469784d758b0ac8884afaf401)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:47.200752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:47.229052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 944/1889] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:47.288865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8051fe9-1ece-482b-8169-5ba77596981c None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:47.291152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e8051fe9-1ece-482b-8169-5ba77596981c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:47.347448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad90ee2469784d758b0ac8884afaf401'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:47.349323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad90ee2469784d758b0ac8884afaf401'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:47.359329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad90ee2469784d758b0ac8884afaf401'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:47.360578 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad90ee2469784d758b0ac8884afaf401'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:47.378784 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad90ee2469784d758b0ac8884afaf401'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:47.428084 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 946/1890] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:46 2022] DELETE /identity/v3/users/ad90ee2469784d758b0ac8884afaf401 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:30:47.500230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8051fe9-1ece-482b-8169-5ba77596981c None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:47.627543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e602a0-df16-462b-a1f5-a82010de493e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:47.629367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e602a0-df16-462b-a1f5-a82010de493e service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:47.629367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e602a0-df16-462b-a1f5-a82010de493e service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:47.629561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e602a0-df16-462b-a1f5-a82010de493e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:47.632614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-40e602a0-df16-462b-a1f5-a82010de493e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:47.760689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e602a0-df16-462b-a1f5-a82010de493e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:47.762879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e602a0-df16-462b-a1f5-a82010de493e service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:47.777264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 945/1891] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:48.241697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:48.243237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:48.266685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c3e132-ea31-4aec-974c-afbde3a25703 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:48.275054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:48.276821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:48.277331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:48.277839 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] PATH_INFO: `/v3/projects/8e1b08d66c6e4b2cb850a3eb972586bb` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:48.280398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e1b08d66c6e4b2cb850a3eb972586bb)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:48.283584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:48.344805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:48.348587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e1b08d66c6e4b2cb850a3eb972586bb', 'request_id': 'req-d6f8feb1-bcb4-4872-840d-2777380eef32'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:48.360175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e1b08d66c6e4b2cb850a3eb972586bb', 'request_id': 'req-d6f8feb1-bcb4-4872-840d-2777380eef32'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:48.370503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e1b08d66c6e4b2cb850a3eb972586bb', 'request_id': 'req-d6f8feb1-bcb4-4872-840d-2777380eef32'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:48.388326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] The token cache is being invalidate because project 8e1b08d66c6e4b2cb850a3eb972586bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:48.388621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:48.389279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d6f8feb1-bcb4-4872-840d-2777380eef32 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:48.391144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 947/1892] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:48 2022] DELETE /identity/v3/projects/8e1b08d66c6e4b2cb850a3eb972586bb => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:48.396426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fe8810-a150-452b-ae0b-f5817586e118 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:48.396714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fe8810-a150-452b-ae0b-f5817586e118 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:48.396991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fe8810-a150-452b-ae0b-f5817586e118 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:48.417775 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e4fe8810-a150-452b-ae0b-f5817586e118 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:48.419038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e4fe8810-a150-452b-ae0b-f5817586e118 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:48.428195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:48.484355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 946/1893] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:48.492218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2739eaef-0c61-45df-8267-02b06e8f3a4e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:48.492503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2739eaef-0c61-45df-8267-02b06e8f3a4e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:48.492778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2739eaef-0c61-45df-8267-02b06e8f3a4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:48.516270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2739eaef-0c61-45df-8267-02b06e8f3a4e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:48.517024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2739eaef-0c61-45df-8267-02b06e8f3a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:48.538429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 948/1894] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:48.543654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fe8810-a150-452b-ae0b-f5817586e118 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:48.554436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fe8810-a150-452b-ae0b-f5817586e118 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:48.556681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4fc52c-603d-42c0-b2ec-1848da0918af None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:48.557548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4fc52c-603d-42c0-b2ec-1848da0918af None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:48.557826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4fc52c-603d-42c0-b2ec-1848da0918af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:48.562391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd99643-dcdc-4bba-8094-bd3259078553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:48.563223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd99643-dcdc-4bba-8094-bd3259078553 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:48.563654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd99643-dcdc-4bba-8094-bd3259078553 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:48.564080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd99643-dcdc-4bba-8094-bd3259078553 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:48.564872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd99643-dcdc-4bba-8094-bd3259078553 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:30:48.565315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd99643-dcdc-4bba-8094-bd3259078553 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:48.567861 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd99643-dcdc-4bba-8094-bd3259078553 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:48.581924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 947/1895] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:30:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:30:48.586225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd99643-dcdc-4bba-8094-bd3259078553 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:48.599660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-dc4fc52c-603d-42c0-b2ec-1848da0918af None None] MFA Rules not processed for user `5ae72d51e9004670a50362d1206fab21`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:48.600879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-dc4fc52c-603d-42c0-b2ec-1848da0918af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:48.607653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd99643-dcdc-4bba-8094-bd3259078553 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:48.613295 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659dc1f8-5b94-4399-b86e-c1cb646f7c38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:48.614725 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659dc1f8-5b94-4399-b86e-c1cb646f7c38 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:48.615184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659dc1f8-5b94-4399-b86e-c1cb646f7c38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:48.615554 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659dc1f8-5b94-4399-b86e-c1cb646f7c38 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:48.616294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659dc1f8-5b94-4399-b86e-c1cb646f7c38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:48.619958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659dc1f8-5b94-4399-b86e-c1cb646f7c38 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:48.673739 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 948/1896] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:48 2022] POST /identity/v3/projects => generated 367 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:30:48.675450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659dc1f8-5b94-4399-b86e-c1cb646f7c38 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:48.706417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 949/1897] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:30:48.724327 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4fc52c-603d-42c0-b2ec-1848da0918af None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:48.726361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-dc4fc52c-603d-42c0-b2ec-1848da0918af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:48.768379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659dc1f8-5b94-4399-b86e-c1cb646f7c38 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:48.773928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c520f3-beab-4608-8033-f12a0c4c02ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:48.776012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c520f3-beab-4608-8033-f12a0c4c02ce admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:48.776405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c520f3-beab-4608-8033-f12a0c4c02ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:48.776775 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c520f3-beab-4608-8033-f12a0c4c02ce admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:48.777670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c520f3-beab-4608-8033-f12a0c4c02ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:48.781822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c520f3-beab-4608-8033-f12a0c4c02ce admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:48.819892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4fc52c-603d-42c0-b2ec-1848da0918af None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:48.837523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 949/1898] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:48 2022] POST /identity/v3/users => generated 284 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:48.839367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c520f3-beab-4608-8033-f12a0c4c02ce admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:48.859958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c520f3-beab-4608-8033-f12a0c4c02ce admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:48.870877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787df605-5082-4557-96a8-5250ec43f897 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:48.871517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787df605-5082-4557-96a8-5250ec43f897 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:48.871801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787df605-5082-4557-96a8-5250ec43f897 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:48.872144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787df605-5082-4557-96a8-5250ec43f897 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:48.873232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787df605-5082-4557-96a8-5250ec43f897 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:48.874945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787df605-5082-4557-96a8-5250ec43f897 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:48.903382 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 950/1899] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:48 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:48.907925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787df605-5082-4557-96a8-5250ec43f897 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:48.918830 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cd7b1c-8d38-479a-8873-5270bffe47c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:48.919954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cd7b1c-8d38-479a-8873-5270bffe47c1 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:48.920929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cd7b1c-8d38-479a-8873-5270bffe47c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:48.921388 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cd7b1c-8d38-479a-8873-5270bffe47c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:48.936895 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787df605-5082-4557-96a8-5250ec43f897 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:48.949896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20c5329-dcc7-4c03-a69d-d17b508662f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:48.950744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20c5329-dcc7-4c03-a69d-d17b508662f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:48.950889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20c5329-dcc7-4c03-a69d-d17b508662f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:48.951143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20c5329-dcc7-4c03-a69d-d17b508662f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:48.951793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20c5329-dcc7-4c03-a69d-d17b508662f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:48.953444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20c5329-dcc7-4c03-a69d-d17b508662f5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:48.961021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:48.993146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 951/1900] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:48 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:48.994368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20c5329-dcc7-4c03-a69d-d17b508662f5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:49.014138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cd7b1c-8d38-479a-8873-5270bffe47c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:49.015638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cd7b1c-8d38-479a-8873-5270bffe47c1 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:49.020982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20c5329-dcc7-4c03-a69d-d17b508662f5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:49.031461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc193ff-1553-45b9-b3a5-fec5122fb485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:49.032208 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc193ff-1553-45b9-b3a5-fec5122fb485 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.034646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc193ff-1553-45b9-b3a5-fec5122fb485 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.034954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc193ff-1553-45b9-b3a5-fec5122fb485 admin admin] PATH_INFO: `/v3/projects/9af6ad7365764a39a87775b62c80cf00/users/ca3f75036f6b428db9da19faf7065eb8/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.052910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 950/1901] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:48 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:30:49.073540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc193ff-1553-45b9-b3a5-fec5122fb485 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9af6ad7365764a39a87775b62c80cf00, user_id=ca3f75036f6b428db9da19faf7065eb8, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:49.078507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc193ff-1553-45b9-b3a5-fec5122fb485 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:49.106447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 952/1902] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:48 2022] PUT /identity/v3/projects/9af6ad7365764a39a87775b62c80cf00/users/ca3f75036f6b428db9da19faf7065eb8/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:49.118464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ead1150-32d6-4d66-851f-f88a0e78cc9e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.119032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ead1150-32d6-4d66-851f-f88a0e78cc9e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.119636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ead1150-32d6-4d66-851f-f88a0e78cc9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.190661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-6ead1150-32d6-4d66-851f-f88a0e78cc9e None None] MFA Rules not processed for user `ca3f75036f6b428db9da19faf7065eb8`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:49.194795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6ead1150-32d6-4d66-851f-f88a0e78cc9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:49.356354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 951/1903] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:49 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:49.356981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079a8373-61c4-49fc-8a72-8a1afc72bf67 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.356981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079a8373-61c4-49fc-8a72-8a1afc72bf67 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.356981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079a8373-61c4-49fc-8a72-8a1afc72bf67 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.358953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 953/1904] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:30:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:30:49.360656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ead1150-32d6-4d66-851f-f88a0e78cc9e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:49.397836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604aa72-ed0d-454a-a2c6-ce2df1c900ca None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.398253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604aa72-ed0d-454a-a2c6-ce2df1c900ca None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.398632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e604aa72-ed0d-454a-a2c6-ce2df1c900ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.459778 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e604aa72-ed0d-454a-a2c6-ce2df1c900ca None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:49.462047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e604aa72-ed0d-454a-a2c6-ce2df1c900ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:49.496643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:49.518749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ead1150-32d6-4d66-851f-f88a0e78cc9e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:49.523616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa083613-4c5b-437b-9134-3bbfc7162574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:49.524364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa083613-4c5b-437b-9134-3bbfc7162574 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.524686 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa083613-4c5b-437b-9134-3bbfc7162574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.524969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa083613-4c5b-437b-9134-3bbfc7162574 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.525737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa083613-4c5b-437b-9134-3bbfc7162574 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:49.529079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa083613-4c5b-437b-9134-3bbfc7162574 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:49.613649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 952/1905] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:49 2022] POST /identity/v3/projects => generated 365 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:49.618160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa083613-4c5b-437b-9134-3bbfc7162574 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:49.621388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 954/1906] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:30:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:49.693641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:49.717506 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e604aa72-ed0d-454a-a2c6-ce2df1c900ca None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:49.718558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa083613-4c5b-437b-9134-3bbfc7162574 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:49.718992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e604aa72-ed0d-454a-a2c6-ce2df1c900ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:49.729774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8bdd60-c892-41e6-9922-fca3cc39f0e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:49.730875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8bdd60-c892-41e6-9922-fca3cc39f0e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.730875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8bdd60-c892-41e6-9922-fca3cc39f0e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.731059 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8bdd60-c892-41e6-9922-fca3cc39f0e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.731608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8bdd60-c892-41e6-9922-fca3cc39f0e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:49.733594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8bdd60-c892-41e6-9922-fca3cc39f0e2 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:49.733967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e604aa72-ed0d-454a-a2c6-ce2df1c900ca None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:49.759950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 953/1907] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:49 2022] POST /identity/v3/users => generated 289 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:49.763931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8bdd60-c892-41e6-9922-fca3cc39f0e2 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:49.774284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8bdd60-c892-41e6-9922-fca3cc39f0e2 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:49.779427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeafbe47-61f3-4d81-af7e-5239175b9334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:49.780023 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeafbe47-61f3-4d81-af7e-5239175b9334 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.780309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeafbe47-61f3-4d81-af7e-5239175b9334 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.780579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeafbe47-61f3-4d81-af7e-5239175b9334 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.781214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeafbe47-61f3-4d81-af7e-5239175b9334 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:49.782856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeafbe47-61f3-4d81-af7e-5239175b9334 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:49.794147 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 954/1908] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:30:49.798341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeafbe47-61f3-4d81-af7e-5239175b9334 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:49.808277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeafbe47-61f3-4d81-af7e-5239175b9334 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:49.808454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb03509-d01e-4a93-82a4-3215da64b2d8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:49.808917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb03509-d01e-4a93-82a4-3215da64b2d8 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.809179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb03509-d01e-4a93-82a4-3215da64b2d8 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.809442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb03509-d01e-4a93-82a4-3215da64b2d8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.813126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1c924b-ea63-411d-95d3-10cc64fd753d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:49.813758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1c924b-ea63-411d-95d3-10cc64fd753d admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.814032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1c924b-ea63-411d-95d3-10cc64fd753d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.814330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1c924b-ea63-411d-95d3-10cc64fd753d admin admin] PATH_INFO: `/v3/projects/1d51dd7fa3f54e3f942c42aca5838002/users/03db94b557cf4ec5bc109194287e5211/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.818293 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb03509-d01e-4a93-82a4-3215da64b2d8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:49.819870 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb03509-d01e-4a93-82a4-3215da64b2d8 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:49.829254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1c924b-ea63-411d-95d3-10cc64fd753d admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d51dd7fa3f54e3f942c42aca5838002, user_id=03db94b557cf4ec5bc109194287e5211, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:49.829959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 955/1909] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:49.832061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1c924b-ea63-411d-95d3-10cc64fd753d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:49.847944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 955/1910] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:49 2022] PUT /identity/v3/projects/1d51dd7fa3f54e3f942c42aca5838002/users/03db94b557cf4ec5bc109194287e5211/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:49.852170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb03509-d01e-4a93-82a4-3215da64b2d8 service nova] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:49.863637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb03509-d01e-4a93-82a4-3215da64b2d8 service nova] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:49.920206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d4f129-d164-48c9-8985-22231568d26c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:49.920825 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4f129-d164-48c9-8985-22231568d26c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.921101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4f129-d164-48c9-8985-22231568d26c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.921348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4f129-d164-48c9-8985-22231568d26c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.922013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d4f129-d164-48c9-8985-22231568d26c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:49.923675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d4f129-d164-48c9-8985-22231568d26c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:49.933439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 956/1911] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:49 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:30:49.937873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1c924b-ea63-411d-95d3-10cc64fd753d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:49.949476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1c924b-ea63-411d-95d3-10cc64fd753d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:49.954854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f4a5d0-eb15-438e-b7d5-ff6cc9086014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:49.955500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f4a5d0-eb15-438e-b7d5-ff6cc9086014 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.955775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f4a5d0-eb15-438e-b7d5-ff6cc9086014 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.956042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f4a5d0-eb15-438e-b7d5-ff6cc9086014 admin admin] PATH_INFO: `/v3/projects/1d51dd7fa3f54e3f942c42aca5838002/users/03db94b557cf4ec5bc109194287e5211/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:49.958191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f4a5d0-eb15-438e-b7d5-ff6cc9086014 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d51dd7fa3f54e3f942c42aca5838002, user_id=03db94b557cf4ec5bc109194287e5211, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:49.960836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f4a5d0-eb15-438e-b7d5-ff6cc9086014 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:49.972309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 956/1912] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:30:49 2022] PUT /identity/v3/projects/1d51dd7fa3f54e3f942c42aca5838002/users/03db94b557cf4ec5bc109194287e5211/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:49.983168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73344d31-32f9-4464-a9f1-13262b54b6d4 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:49.983451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73344d31-32f9-4464-a9f1-13262b54b6d4 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:49.983730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73344d31-32f9-4464-a9f1-13262b54b6d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:50.018014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-73344d31-32f9-4464-a9f1-13262b54b6d4 None None] MFA Rules not processed for user `03db94b557cf4ec5bc109194287e5211`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:50.019200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-73344d31-32f9-4464-a9f1-13262b54b6d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:50.043654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:50.095690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 957/1913] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:49 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:50.108521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f4a5d0-eb15-438e-b7d5-ff6cc9086014 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:50.163969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f4a5d0-eb15-438e-b7d5-ff6cc9086014 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:50.168075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0648f4bc-0c48-4672-a829-02eeb1d22f7b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:50.168672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0648f4bc-0c48-4672-a829-02eeb1d22f7b service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:50.168944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0648f4bc-0c48-4672-a829-02eeb1d22f7b service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:50.169208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0648f4bc-0c48-4672-a829-02eeb1d22f7b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:50.178502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0648f4bc-0c48-4672-a829-02eeb1d22f7b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:50.179946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0648f4bc-0c48-4672-a829-02eeb1d22f7b service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:50.189495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 957/1914] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:50 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:50.278772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73344d31-32f9-4464-a9f1-13262b54b6d4 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:50.280303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-73344d31-32f9-4464-a9f1-13262b54b6d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:50.292927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73344d31-32f9-4464-a9f1-13262b54b6d4 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:50.297590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fe3bd4-acb9-4394-a7b4-9bcef1f08af8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:50.298294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fe3bd4-acb9-4394-a7b4-9bcef1f08af8 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:50.298550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fe3bd4-acb9-4394-a7b4-9bcef1f08af8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:50.298815 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fe3bd4-acb9-4394-a7b4-9bcef1f08af8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:50.357654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fe3bd4-acb9-4394-a7b4-9bcef1f08af8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:50.359108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fe3bd4-acb9-4394-a7b4-9bcef1f08af8 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:50.371143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 958/1915] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:50 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:54.989950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab343d3-5f72-40af-af95-2d8f2110535d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:54.991357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab343d3-5f72-40af-af95-2d8f2110535d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:54.991357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab343d3-5f72-40af-af95-2d8f2110535d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:55.032363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-5ab343d3-5f72-40af-af95-2d8f2110535d None None] MFA Rules not processed for user `5ae72d51e9004670a50362d1206fab21`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:55.033991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-5ab343d3-5f72-40af-af95-2d8f2110535d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:55.143700 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 958/1916] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:54 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:57.194282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fe3bd4-acb9-4394-a7b4-9bcef1f08af8 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:57.196292 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d0fe3bd4-acb9-4394-a7b4-9bcef1f08af8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:57.289072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fe3bd4-acb9-4394-a7b4-9bcef1f08af8 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:57.294682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:57.295341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:57.295626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:57.295900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] PATH_INFO: `/v3/users/29157c390595428a9f3803ef341f1e29` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:57.297241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=29157c390595428a9f3803ef341f1e29)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:57.299257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:57.358154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29157c390595428a9f3803ef341f1e29'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:57.367499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29157c390595428a9f3803ef341f1e29'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:57.368286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29157c390595428a9f3803ef341f1e29'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:57.369727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29157c390595428a9f3803ef341f1e29'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:57.376067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:57.378808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29157c390595428a9f3803ef341f1e29'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:57.424487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 959/1917] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:57 2022] DELETE /identity/v3/users/29157c390595428a9f3803ef341f1e29 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:57.429215 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260a135e-b23d-483a-87cd-144f26ed9aac None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:57.429741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260a135e-b23d-483a-87cd-144f26ed9aac None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:57.429818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260a135e-b23d-483a-87cd-144f26ed9aac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:57.467302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-260a135e-b23d-483a-87cd-144f26ed9aac None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:57.468616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-260a135e-b23d-483a-87cd-144f26ed9aac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:57.496570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:57.532047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 959/1918] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:57.542263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:57.543723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:57.616487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31645f18-dbaa-4ab8-9a69-f6beeff40ce4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:57.623988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce633f9-17c7-40d8-962d-805710d0fd7c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:57.624741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce633f9-17c7-40d8-962d-805710d0fd7c service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:57.625082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce633f9-17c7-40d8-962d-805710d0fd7c service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:57.625423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce633f9-17c7-40d8-962d-805710d0fd7c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:57.635843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce633f9-17c7-40d8-962d-805710d0fd7c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:57.637322 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce633f9-17c7-40d8-962d-805710d0fd7c service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:57.648571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 960/1919] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:57.919742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260a135e-b23d-483a-87cd-144f26ed9aac None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:57.921158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-260a135e-b23d-483a-87cd-144f26ed9aac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:57.943413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260a135e-b23d-483a-87cd-144f26ed9aac None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:57.950925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:57.951852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:57.952260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:57.952656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] PATH_INFO: `/v3/projects/8b934b7f515d446786fc24961906d672` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:57.953946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b934b7f515d446786fc24961906d672)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:57.956064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:58.014262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b934b7f515d446786fc24961906d672', 'request_id': 'req-76c80805-ff93-4a83-994b-bb833ad83b92'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.025453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b934b7f515d446786fc24961906d672', 'request_id': 'req-76c80805-ff93-4a83-994b-bb833ad83b92'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.037016 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b934b7f515d446786fc24961906d672', 'request_id': 'req-76c80805-ff93-4a83-994b-bb833ad83b92'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.049873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] The token cache is being invalidate because project 8b934b7f515d446786fc24961906d672 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:58.050136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.051573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.053209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 960/1920] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:57 2022] DELETE /identity/v3/projects/8b934b7f515d446786fc24961906d672 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:58.057342 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce633f9-17c7-40d8-962d-805710d0fd7c service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:58.067300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce633f9-17c7-40d8-962d-805710d0fd7c service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:58.134381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:58.135065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:58.135341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:58.135604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] PATH_INFO: `/v3/users/25dc9fc5ed4040f88edb5b56c80bd5f7` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:58.136742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25dc9fc5ed4040f88edb5b56c80bd5f7)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:58.138691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:58.189787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25dc9fc5ed4040f88edb5b56c80bd5f7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.203600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25dc9fc5ed4040f88edb5b56c80bd5f7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.204605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25dc9fc5ed4040f88edb5b56c80bd5f7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.205297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25dc9fc5ed4040f88edb5b56c80bd5f7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.214299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-333cda2a-936d-4dfc-b52e-891942ff6621 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25dc9fc5ed4040f88edb5b56c80bd5f7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.246533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:58.259620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 961/1921] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:58 2022] DELETE /identity/v3/users/25dc9fc5ed4040f88edb5b56c80bd5f7 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:58.290072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:58.292346 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:58.308584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c80805-ff93-4a83-994b-bb833ad83b92 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:58.388335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:58.389248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:58.389867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:58.390480 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] PATH_INFO: `/v3/users/ca3f75036f6b428db9da19faf7065eb8` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:58.391801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca3f75036f6b428db9da19faf7065eb8)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:58.393981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:58.454952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca3f75036f6b428db9da19faf7065eb8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.455642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca3f75036f6b428db9da19faf7065eb8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.467093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca3f75036f6b428db9da19faf7065eb8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.468506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca3f75036f6b428db9da19faf7065eb8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.470704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:58.480623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca3f75036f6b428db9da19faf7065eb8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.503187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 961/1922] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:58 2022] DELETE /identity/v3/users/ca3f75036f6b428db9da19faf7065eb8 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:58.508392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff7615-eabb-48de-9fb9-5700207cdd62 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:58.508392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff7615-eabb-48de-9fb9-5700207cdd62 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:58.508618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fff7615-eabb-48de-9fb9-5700207cdd62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:58.538040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-1fff7615-eabb-48de-9fb9-5700207cdd62 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:58.539318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1fff7615-eabb-48de-9fb9-5700207cdd62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:58.618784 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 962/1923] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:58.629877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:58.631352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:58.657604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff7615-eabb-48de-9fb9-5700207cdd62 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:58.659498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1fff7615-eabb-48de-9fb9-5700207cdd62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:58.677462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fff7615-eabb-48de-9fb9-5700207cdd62 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:58.691555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:58.692400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:58.692685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:58.692946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] PATH_INFO: `/v3/projects/e5a6119b7301452bb4181c5b9481c7a9` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:58.700514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5a6119b7301452bb4181c5b9481c7a9)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:58.702625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:58.761355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e7cb5c-b8cf-4cf9-8c3e-832eace2ed6c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:58.765261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cea4e5-4b10-41d6-b92b-fcb417267995 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:58.766562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cea4e5-4b10-41d6-b92b-fcb417267995 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:58.766562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cea4e5-4b10-41d6-b92b-fcb417267995 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:58.766562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cea4e5-4b10-41d6-b92b-fcb417267995 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:58.781282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cea4e5-4b10-41d6-b92b-fcb417267995 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:58.783144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cea4e5-4b10-41d6-b92b-fcb417267995 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:58.786057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5a6119b7301452bb4181c5b9481c7a9', 'request_id': 'req-773c5787-576d-4aa8-ab56-0a221b64cb57'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.794342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 962/1924] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:30:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:30:58.795002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5a6119b7301452bb4181c5b9481c7a9', 'request_id': 'req-773c5787-576d-4aa8-ab56-0a221b64cb57'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.804854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5a6119b7301452bb4181c5b9481c7a9', 'request_id': 'req-773c5787-576d-4aa8-ab56-0a221b64cb57'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.819722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] The token cache is being invalidate because project e5a6119b7301452bb4181c5b9481c7a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:58.820064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.820722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-773c5787-576d-4aa8-ab56-0a221b64cb57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:58.822956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 963/1925] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:58 2022] DELETE /identity/v3/projects/e5a6119b7301452bb4181c5b9481c7a9 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:30:58.827554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8f824-d435-40c2-9c43-11a0872f69bb None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:58.828288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8f824-d435-40c2-9c43-11a0872f69bb None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:58.828288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8f824-d435-40c2-9c43-11a0872f69bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:58.852474 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-47d8f824-d435-40c2-9c43-11a0872f69bb None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:58.853740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-47d8f824-d435-40c2-9c43-11a0872f69bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:58.983672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 963/1926] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:58.991543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd0875b-b66c-46ec-9b30-9302191a2e7b None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:58.992420 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd0875b-b66c-46ec-9b30-9302191a2e7b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:58.992798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd0875b-b66c-46ec-9b30-9302191a2e7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:59.026232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-bcd0875b-b66c-46ec-9b30-9302191a2e7b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:59.026232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-bcd0875b-b66c-46ec-9b30-9302191a2e7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:59.050165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 964/1927] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:30:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:59.054740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d8f824-d435-40c2-9c43-11a0872f69bb None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:59.099661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d8f824-d435-40c2-9c43-11a0872f69bb None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:59.113161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3562007a-8567-47fa-8f07-2b0714a6db80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:59.114426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3562007a-8567-47fa-8f07-2b0714a6db80 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:59.114827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3562007a-8567-47fa-8f07-2b0714a6db80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:59.115190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3562007a-8567-47fa-8f07-2b0714a6db80 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:59.116512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3562007a-8567-47fa-8f07-2b0714a6db80 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:30:59.116634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3562007a-8567-47fa-8f07-2b0714a6db80 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:59.119230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3562007a-8567-47fa-8f07-2b0714a6db80 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:59.138029 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 964/1928] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:30:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:30:59.145026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd0875b-b66c-46ec-9b30-9302191a2e7b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:59.160131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd0875b-b66c-46ec-9b30-9302191a2e7b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:59.173425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44706aa0-a7b7-4dc8-b326-d3d0a561f7a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:59.176266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44706aa0-a7b7-4dc8-b326-d3d0a561f7a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:59.176646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44706aa0-a7b7-4dc8-b326-d3d0a561f7a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:59.179245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44706aa0-a7b7-4dc8-b326-d3d0a561f7a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:59.180034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44706aa0-a7b7-4dc8-b326-d3d0a561f7a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:59.185814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44706aa0-a7b7-4dc8-b326-d3d0a561f7a0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:59.255866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 965/1929] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:30:59 2022] POST /identity/v3/projects => generated 371 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:59.258922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3562007a-8567-47fa-8f07-2b0714a6db80 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:59.279426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3562007a-8567-47fa-8f07-2b0714a6db80 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:59.379373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439a5498-c685-4eb6-98d9-5dd4129be713 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:59.379673 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439a5498-c685-4eb6-98d9-5dd4129be713 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:59.380004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439a5498-c685-4eb6-98d9-5dd4129be713 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:59.381472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 966/1930] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:30:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:30:59.403051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439a5498-c685-4eb6-98d9-5dd4129be713 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:59.406603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-439a5498-c685-4eb6-98d9-5dd4129be713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:59.588680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:59.602753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0da9ea-e91d-4303-99e5-5d8bbcd88391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:59.603749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0da9ea-e91d-4303-99e5-5d8bbcd88391 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:59.604236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0da9ea-e91d-4303-99e5-5d8bbcd88391 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:59.604692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0da9ea-e91d-4303-99e5-5d8bbcd88391 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:59.606003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439a5498-c685-4eb6-98d9-5dd4129be713 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:59.609496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0da9ea-e91d-4303-99e5-5d8bbcd88391 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:59.613600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0da9ea-e91d-4303-99e5-5d8bbcd88391 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:59.626920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:59.628326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:59.628696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:59.629589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] PATH_INFO: `/v3/projects/9af6ad7365764a39a87775b62c80cf00` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:59.630816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=3,keystone:DELETE=3,keystone:SELECT=94 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:30:59.631405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] RBAC: Authorizing `identity:delete_project(project_id=9af6ad7365764a39a87775b62c80cf00)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:30:59.639435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:30:59.657730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 965/1931] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:30:59 2022] POST /identity/v3/users => generated 286 bytes in 397 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:30:59.657730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c98402-e061-4470-880d-a8e9fdf1185b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:59.657730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c98402-e061-4470-880d-a8e9fdf1185b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:59.657730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c98402-e061-4470-880d-a8e9fdf1185b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:59.705116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-a2c98402-e061-4470-880d-a8e9fdf1185b None None] MFA Rules not processed for user `4bc6be7644b24ca9bab23128b41a7148`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:30:59.709061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a2c98402-e061-4470-880d-a8e9fdf1185b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:59.765299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9af6ad7365764a39a87775b62c80cf00', 'request_id': 'req-f0e8affe-c086-4f6b-a0e0-2572d45d326c'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:59.775804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9af6ad7365764a39a87775b62c80cf00', 'request_id': 'req-f0e8affe-c086-4f6b-a0e0-2572d45d326c'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:59.795839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9af6ad7365764a39a87775b62c80cf00', 'request_id': 'req-f0e8affe-c086-4f6b-a0e0-2572d45d326c'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:59.821818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] The token cache is being invalidate because project 9af6ad7365764a39a87775b62c80cf00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:30:59.822210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:59.824972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:30:59.830939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 967/1932] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:30:59 2022] DELETE /identity/v3/projects/9af6ad7365764a39a87775b62c80cf00 => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:30:59.834477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:59.835017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 966/1933] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:30:59 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:30:59.835946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:59.836124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c98402-e061-4470-880d-a8e9fdf1185b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:30:59.841834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a2c98402-e061-4470-880d-a8e9fdf1185b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:30:59.869285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c98402-e061-4470-880d-a8e9fdf1185b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:30:59.995140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:30:59.995953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:30:59.996348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:30:59.996689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] PATH_INFO: `/v3/users/03db94b557cf4ec5bc109194287e5211` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:30:59.999496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=03db94b557cf4ec5bc109194287e5211)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:00.004512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:00.027258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e8affe-c086-4f6b-a0e0-2572d45d326c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:00.043123 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79520c31-6e5c-4c35-b103-b794d874a414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:00.043774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79520c31-6e5c-4c35-b103-b794d874a414 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:00.044067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79520c31-6e5c-4c35-b103-b794d874a414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:00.044359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79520c31-6e5c-4c35-b103-b794d874a414 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:00.045078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79520c31-6e5c-4c35-b103-b794d874a414 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:00.048618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79520c31-6e5c-4c35-b103-b794d874a414 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:00.081600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 968/1934] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:30:59 2022] GET /identity/v3/roles => generated 1420 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:00.082205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79520c31-6e5c-4c35-b103-b794d874a414 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:00.084366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-79520c31-6e5c-4c35-b103-b794d874a414 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:00.130510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03db94b557cf4ec5bc109194287e5211'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:00.131464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03db94b557cf4ec5bc109194287e5211'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:00.143942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03db94b557cf4ec5bc109194287e5211'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:00.144899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03db94b557cf4ec5bc109194287e5211'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:00.165456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03db94b557cf4ec5bc109194287e5211'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:00.196881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 967/1935] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:30:59 2022] DELETE /identity/v3/users/03db94b557cf4ec5bc109194287e5211 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:00.198480 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:00.200606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:00.209826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79520c31-6e5c-4c35-b103-b794d874a414 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:00.225114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:00.240431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:00.311091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3845c457-1d67-4857-b568-4f506d49ccf6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:00.312567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3845c457-1d67-4857-b568-4f506d49ccf6 service placement] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:00.312567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3845c457-1d67-4857-b568-4f506d49ccf6 service placement] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:00.312567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3845c457-1d67-4857-b568-4f506d49ccf6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:00.314814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3845c457-1d67-4857-b568-4f506d49ccf6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:00.330374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3845c457-1d67-4857-b568-4f506d49ccf6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:00.333126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3845c457-1d67-4857-b568-4f506d49ccf6 service placement] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:00.346061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ec567b-812b-4e9f-9496-b82897727c1d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:00.351764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b779694-a79b-48a4-ba34-b6c846baa2c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:00.352714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b779694-a79b-48a4-ba34-b6c846baa2c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:00.353318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b779694-a79b-48a4-ba34-b6c846baa2c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:00.353772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b779694-a79b-48a4-ba34-b6c846baa2c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:00.354608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b779694-a79b-48a4-ba34-b6c846baa2c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:00.356411 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b779694-a79b-48a4-ba34-b6c846baa2c1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:00.360750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 969/1936] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:00.383512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 968/1937] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:00 2022] GET /identity/v3/roles => generated 1420 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:00.388562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3845c457-1d67-4857-b568-4f506d49ccf6 service placement] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:00.404770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3845c457-1d67-4857-b568-4f506d49ccf6 service placement] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:00.418552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41298488-b95a-488a-a18e-1742f82f6b59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:00.419339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41298488-b95a-488a-a18e-1742f82f6b59 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:00.420344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41298488-b95a-488a-a18e-1742f82f6b59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:00.421237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41298488-b95a-488a-a18e-1742f82f6b59 admin admin] PATH_INFO: `/v3/projects/0c606390d99e45faa1970a54148a51f2/users/d58bde7cde58465d893d26d99893bbe5/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:00.450465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41298488-b95a-488a-a18e-1742f82f6b59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c606390d99e45faa1970a54148a51f2, user_id=d58bde7cde58465d893d26d99893bbe5, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:00.453398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41298488-b95a-488a-a18e-1742f82f6b59 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:00.478413 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 970/1938] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:00 2022] PUT /identity/v3/projects/0c606390d99e45faa1970a54148a51f2/users/d58bde7cde58465d893d26d99893bbe5/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:00.489660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fc7318-b581-4862-aabc-bc1c1d2e67d0 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:00.489940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fc7318-b581-4862-aabc-bc1c1d2e67d0 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:00.490244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fc7318-b581-4862-aabc-bc1c1d2e67d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:00.544345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-74fc7318-b581-4862-aabc-bc1c1d2e67d0 None None] MFA Rules not processed for user `d58bde7cde58465d893d26d99893bbe5`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:00.546013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-74fc7318-b581-4862-aabc-bc1c1d2e67d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:00.636020 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41298488-b95a-488a-a18e-1742f82f6b59 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:00.636626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 969/1939] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:00 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:00.692740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fc7318-b581-4862-aabc-bc1c1d2e67d0 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:00.695662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-74fc7318-b581-4862-aabc-bc1c1d2e67d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:00.767737 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41298488-b95a-488a-a18e-1742f82f6b59 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:00.774178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4697d233-81a7-499c-a694-e9265da540e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:00.775173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4697d233-81a7-499c-a694-e9265da540e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:00.775541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4697d233-81a7-499c-a694-e9265da540e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:00.776039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4697d233-81a7-499c-a694-e9265da540e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:00.776773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4697d233-81a7-499c-a694-e9265da540e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:00.780002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4697d233-81a7-499c-a694-e9265da540e4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:00.783708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fc7318-b581-4862-aabc-bc1c1d2e67d0 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:00.789124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:00.789771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:00.790054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:00.790386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] PATH_INFO: `/v3/users/c9a79828d83e4535bb46631b52b7d7ec` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:00.791570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9a79828d83e4535bb46631b52b7d7ec)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:00.793601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:00.817829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 971/1940] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:00 2022] POST /identity/v3/projects => generated 371 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:00.818644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4697d233-81a7-499c-a694-e9265da540e4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:00.820230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4697d233-81a7-499c-a694-e9265da540e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:00.844187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:00.858763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9a79828d83e4535bb46631b52b7d7ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:00.859596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9a79828d83e4535bb46631b52b7d7ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:00.869235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9a79828d83e4535bb46631b52b7d7ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:00.870116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9a79828d83e4535bb46631b52b7d7ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:00.881230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9a79828d83e4535bb46631b52b7d7ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:00.901627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 970/1941] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:00 2022] DELETE /identity/v3/users/c9a79828d83e4535bb46631b52b7d7ec => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:00.902593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:00.903974 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:00.905366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4697d233-81a7-499c-a694-e9265da540e4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:00.924718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:01.007109 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.007867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.008212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.008901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] PATH_INFO: `/v3/projects/1d51dd7fa3f54e3f942c42aca5838002` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.010352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d51dd7fa3f54e3f942c42aca5838002)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.012380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.018048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab492f28-f20f-4599-babc-663b8240c313 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.028986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb811a00-8dd4-45d0-8b0e-b68db4bea73c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.030807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb811a00-8dd4-45d0-8b0e-b68db4bea73c admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.032762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb811a00-8dd4-45d0-8b0e-b68db4bea73c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.033251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb811a00-8dd4-45d0-8b0e-b68db4bea73c admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.035860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb811a00-8dd4-45d0-8b0e-b68db4bea73c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.041563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb811a00-8dd4-45d0-8b0e-b68db4bea73c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.074402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 971/1942] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:00 2022] POST /identity/v3/users => generated 292 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:01.077001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb811a00-8dd4-45d0-8b0e-b68db4bea73c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.079068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d51dd7fa3f54e3f942c42aca5838002', 'request_id': 'req-e327033e-e941-4eb8-a2b3-c5126dfbb324'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.087393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d51dd7fa3f54e3f942c42aca5838002', 'request_id': 'req-e327033e-e941-4eb8-a2b3-c5126dfbb324'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.090541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb811a00-8dd4-45d0-8b0e-b68db4bea73c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.099456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d51dd7fa3f54e3f942c42aca5838002', 'request_id': 'req-e327033e-e941-4eb8-a2b3-c5126dfbb324'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.115082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] The token cache is being invalidate because project 1d51dd7fa3f54e3f942c42aca5838002 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:01.115481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.116190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e327033e-e941-4eb8-a2b3-c5126dfbb324 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.118539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 972/1943] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:00 2022] DELETE /identity/v3/projects/1d51dd7fa3f54e3f942c42aca5838002 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:01.128030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842b6f8f-f422-474e-b41c-ed5076c1f665 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.128330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842b6f8f-f422-474e-b41c-ed5076c1f665 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.128857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842b6f8f-f422-474e-b41c-ed5076c1f665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.170921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-842b6f8f-f422-474e-b41c-ed5076c1f665 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:01.172913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-842b6f8f-f422-474e-b41c-ed5076c1f665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:01.197839 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bafedf-cd34-4cb8-a423-037780bf0a41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.198843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bafedf-cd34-4cb8-a423-037780bf0a41 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.199126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bafedf-cd34-4cb8-a423-037780bf0a41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.199401 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bafedf-cd34-4cb8-a423-037780bf0a41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.200050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bafedf-cd34-4cb8-a423-037780bf0a41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.201676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bafedf-cd34-4cb8-a423-037780bf0a41 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.213438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 972/1944] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:01 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:01.214453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bafedf-cd34-4cb8-a423-037780bf0a41 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.219041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-41bafedf-cd34-4cb8-a423-037780bf0a41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:01.233595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bafedf-cd34-4cb8-a423-037780bf0a41 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.238299 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.238927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.239202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.239472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] PATH_INFO: `/v3/projects/4e70bc4c4f6c429daf16241a82624849` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.240568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e70bc4c4f6c429daf16241a82624849)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.242524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 973/1945] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:01.242713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.243346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842b6f8f-f422-474e-b41c-ed5076c1f665 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.244917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-842b6f8f-f422-474e-b41c-ed5076c1f665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:01.257014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842b6f8f-f422-474e-b41c-ed5076c1f665 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.262532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d253e136-24f6-4ec7-a4ae-75ddc2b82ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.263193 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d253e136-24f6-4ec7-a4ae-75ddc2b82ff8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.263464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d253e136-24f6-4ec7-a4ae-75ddc2b82ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.263726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d253e136-24f6-4ec7-a4ae-75ddc2b82ff8 admin admin] PATH_INFO: `/v3/projects/b8e44671811d4b228bb4bd334df5cd43/users/b2f3818017324197b9b7143fb2fa8d25/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.279691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d253e136-24f6-4ec7-a4ae-75ddc2b82ff8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8e44671811d4b228bb4bd334df5cd43, user_id=b2f3818017324197b9b7143fb2fa8d25, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.281765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:01.282766 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d253e136-24f6-4ec7-a4ae-75ddc2b82ff8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.293692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e70bc4c4f6c429daf16241a82624849', 'request_id': 'req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.297950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 974/1946] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:01 2022] PUT /identity/v3/projects/b8e44671811d4b228bb4bd334df5cd43/users/b2f3818017324197b9b7143fb2fa8d25/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:01.299302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd62876-f244-4624-bed9-1e543b2c5da2 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.299718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd62876-f244-4624-bed9-1e543b2c5da2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.299996 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd62876-f244-4624-bed9-1e543b2c5da2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.303028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e70bc4c4f6c429daf16241a82624849', 'request_id': 'req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.311318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e70bc4c4f6c429daf16241a82624849', 'request_id': 'req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.319073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4dd62876-f244-4624-bed9-1e543b2c5da2 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:01.320278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4dd62876-f244-4624-bed9-1e543b2c5da2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:01.323373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] The token cache is being invalidate because project 4e70bc4c4f6c429daf16241a82624849 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:01.323658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.324300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.328780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 973/1947] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:01 2022] DELETE /identity/v3/projects/4e70bc4c4f6c429daf16241a82624849 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:01.328780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.328780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:01.350151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:01.387210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 975/1948] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:01.388113 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd62876-f244-4624-bed9-1e543b2c5da2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.389457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4dd62876-f244-4624-bed9-1e543b2c5da2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:01.395687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108f892e-7fe8-4a1d-92c5-03e9d2ba4000 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.400793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca44d8e-ae46-44da-8408-dc673fd214ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.400933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd62876-f244-4624-bed9-1e543b2c5da2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.401412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca44d8e-ae46-44da-8408-dc673fd214ae admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.401697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca44d8e-ae46-44da-8408-dc673fd214ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.401995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca44d8e-ae46-44da-8408-dc673fd214ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.402676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca44d8e-ae46-44da-8408-dc673fd214ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.404326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca44d8e-ae46-44da-8408-dc673fd214ae admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.405419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.406032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.406328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.406614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] PATH_INFO: `/v3/users/a9040b0c066142438efa9a9e603e4ed0` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.407728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=a9040b0c066142438efa9a9e603e4ed0)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.409619 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.414492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 974/1949] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:01 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:01.415398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca44d8e-ae46-44da-8408-dc673fd214ae admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.426744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca44d8e-ae46-44da-8408-dc673fd214ae admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.431359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3818579-0b9e-4021-9bea-b2a997a5eb04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.431963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3818579-0b9e-4021-9bea-b2a997a5eb04 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.432249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3818579-0b9e-4021-9bea-b2a997a5eb04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.432516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3818579-0b9e-4021-9bea-b2a997a5eb04 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.433159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3818579-0b9e-4021-9bea-b2a997a5eb04 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:01.433460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3818579-0b9e-4021-9bea-b2a997a5eb04 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.435096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3818579-0b9e-4021-9bea-b2a997a5eb04 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.449163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 975/1950] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:01.450096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3818579-0b9e-4021-9bea-b2a997a5eb04 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.461030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a9040b0c066142438efa9a9e603e4ed0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.461801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3818579-0b9e-4021-9bea-b2a997a5eb04 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.466568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1659fc8f-706e-4f67-aa16-7cc08ee58814 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.467786 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1659fc8f-706e-4f67-aa16-7cc08ee58814 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.468046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1659fc8f-706e-4f67-aa16-7cc08ee58814 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.468312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1659fc8f-706e-4f67-aa16-7cc08ee58814 admin admin] PATH_INFO: `/v3/projects/b8e44671811d4b228bb4bd334df5cd43/users/b2f3818017324197b9b7143fb2fa8d25/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.469622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a9040b0c066142438efa9a9e603e4ed0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.470288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a9040b0c066142438efa9a9e603e4ed0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.470465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1659fc8f-706e-4f67-aa16-7cc08ee58814 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8e44671811d4b228bb4bd334df5cd43, user_id=b2f3818017324197b9b7143fb2fa8d25, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.470952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a9040b0c066142438efa9a9e603e4ed0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.474821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1659fc8f-706e-4f67-aa16-7cc08ee58814 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.479610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bfaf68f3-685f-45f9-9508-75d0558ca8cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a9040b0c066142438efa9a9e603e4ed0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:01.497160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 976/1951] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:01 2022] PUT /identity/v3/projects/b8e44671811d4b228bb4bd334df5cd43/users/b2f3818017324197b9b7143fb2fa8d25/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:01.497160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1659fc8f-706e-4f67-aa16-7cc08ee58814 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.498540 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1659fc8f-706e-4f67-aa16-7cc08ee58814 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:01.510976 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 976/1952] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:01 2022] DELETE /identity/v3/users/a9040b0c066142438efa9a9e603e4ed0 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:01.512643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a93e54-beb3-4c1c-a423-3b2fc111003b None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.512980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a93e54-beb3-4c1c-a423-3b2fc111003b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.513310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a93e54-beb3-4c1c-a423-3b2fc111003b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.574313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-62a93e54-beb3-4c1c-a423-3b2fc111003b None None] MFA Rules not processed for user `b2f3818017324197b9b7143fb2fa8d25`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:01.575862 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-62a93e54-beb3-4c1c-a423-3b2fc111003b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:01.607058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1659fc8f-706e-4f67-aa16-7cc08ee58814 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.665359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 977/1953] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:01 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:01.673668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:DELETE=5,keystone:SELECT=93,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:01.679631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a93e54-beb3-4c1c-a423-3b2fc111003b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.682080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-62a93e54-beb3-4c1c-a423-3b2fc111003b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:01.725312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ad1e5e-9dde-49fb-99d9-f1547e9717f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.726247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ad1e5e-9dde-49fb-99d9-f1547e9717f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.726696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ad1e5e-9dde-49fb-99d9-f1547e9717f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.727116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ad1e5e-9dde-49fb-99d9-f1547e9717f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.727923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ad1e5e-9dde-49fb-99d9-f1547e9717f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.734816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ad1e5e-9dde-49fb-99d9-f1547e9717f8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.753591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:01.776915 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 977/1954] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:01 2022] POST /identity/v3/projects => generated 359 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:01.781289 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ad1e5e-9dde-49fb-99d9-f1547e9717f8 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.816841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a93e54-beb3-4c1c-a423-3b2fc111003b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.903442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3ef511-1bec-4b77-b85a-75a0ff81e9f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.904177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3ef511-1bec-4b77-b85a-75a0ff81e9f3 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.904513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3ef511-1bec-4b77-b85a-75a0ff81e9f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.904756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3ef511-1bec-4b77-b85a-75a0ff81e9f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.928940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ad1e5e-9dde-49fb-99d9-f1547e9717f8 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:01.936017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744cc69a-6548-4652-84a0-3a1a0869e902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:01.936656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744cc69a-6548-4652-84a0-3a1a0869e902 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:01.936934 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744cc69a-6548-4652-84a0-3a1a0869e902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:01.937204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744cc69a-6548-4652-84a0-3a1a0869e902 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:01.937858 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744cc69a-6548-4652-84a0-3a1a0869e902 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:01.939877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744cc69a-6548-4652-84a0-3a1a0869e902 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:01.987778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 978/1955] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:01 2022] POST /identity/v3/users => generated 280 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:31:01.989145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744cc69a-6548-4652-84a0-3a1a0869e902 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:01.991596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-744cc69a-6548-4652-84a0-3a1a0869e902 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.002286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3ef511-1bec-4b77-b85a-75a0ff81e9f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.003904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3ef511-1bec-4b77-b85a-75a0ff81e9f3 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.012209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744cc69a-6548-4652-84a0-3a1a0869e902 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.017831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.018878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.019160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.019437 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] PATH_INFO: `/v3/projects/b9231b54e2b447bdba74d6282decf896` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.020605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9231b54e2b447bdba74d6282decf896)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.022235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 978/1956] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:01 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:02.022655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.023723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3ef511-1bec-4b77-b85a-75a0ff81e9f3 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.038231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3ef511-1bec-4b77-b85a-75a0ff81e9f3 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.044367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd0c875-0023-43bb-b52d-978962b21a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.046075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd0c875-0023-43bb-b52d-978962b21a59 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.046599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd0c875-0023-43bb-b52d-978962b21a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.047061 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd0c875-0023-43bb-b52d-978962b21a59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.047950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd0c875-0023-43bb-b52d-978962b21a59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.050540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd0c875-0023-43bb-b52d-978962b21a59 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.072508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 979/1957] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:02 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:02.073640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd0c875-0023-43bb-b52d-978962b21a59 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.085433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9231b54e2b447bdba74d6282decf896', 'request_id': 'req-086fa6ab-0638-42d0-a63f-a02bb0e167b4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.085973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd0c875-0023-43bb-b52d-978962b21a59 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.092778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9231b54e2b447bdba74d6282decf896', 'request_id': 'req-086fa6ab-0638-42d0-a63f-a02bb0e167b4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.116929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9231b54e2b447bdba74d6282decf896', 'request_id': 'req-086fa6ab-0638-42d0-a63f-a02bb0e167b4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.135307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] The token cache is being invalidate because project b9231b54e2b447bdba74d6282decf896 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:02.135660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.138514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-086fa6ab-0638-42d0-a63f-a02bb0e167b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.141407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 979/1958] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:01 2022] DELETE /identity/v3/projects/b9231b54e2b447bdba74d6282decf896 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:02.146372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4ac0a1-1b29-4116-a67a-1bb17c024a56 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.147094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4ac0a1-1b29-4116-a67a-1bb17c024a56 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.147794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4ac0a1-1b29-4116-a67a-1bb17c024a56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.179527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-9f4ac0a1-1b29-4116-a67a-1bb17c024a56 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:02.180336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9f4ac0a1-1b29-4116-a67a-1bb17c024a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.188852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829c4a2a-4cf4-4c41-bad4-ed2f7473045d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.189528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829c4a2a-4cf4-4c41-bad4-ed2f7473045d admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.189813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829c4a2a-4cf4-4c41-bad4-ed2f7473045d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.190087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829c4a2a-4cf4-4c41-bad4-ed2f7473045d admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.191331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829c4a2a-4cf4-4c41-bad4-ed2f7473045d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.194139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829c4a2a-4cf4-4c41-bad4-ed2f7473045d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.208682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 980/1959] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:02.210438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872d1316-e26b-4b6c-954e-47e554d5afca None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.210720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872d1316-e26b-4b6c-954e-47e554d5afca None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.210992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872d1316-e26b-4b6c-954e-47e554d5afca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.211545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 980/1960] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:02 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:02.218078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829c4a2a-4cf4-4c41-bad4-ed2f7473045d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.223828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-829c4a2a-4cf4-4c41-bad4-ed2f7473045d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.239509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-872d1316-e26b-4b6c-954e-47e554d5afca None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:02.241048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-872d1316-e26b-4b6c-954e-47e554d5afca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.249753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 981/1961] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:02.254029 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872d1316-e26b-4b6c-954e-47e554d5afca None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.255639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829c4a2a-4cf4-4c41-bad4-ed2f7473045d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.260942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28c8ddb-fc50-45a7-bb65-c3ab5866a6ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.261876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28c8ddb-fc50-45a7-bb65-c3ab5866a6ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.262187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28c8ddb-fc50-45a7-bb65-c3ab5866a6ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.262533 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28c8ddb-fc50-45a7-bb65-c3ab5866a6ad admin admin] PATH_INFO: `/v3/projects/0b8db4f8b50e496892d02241083f14f7/users/dd29e17afa534affa9f510a39b869f73/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.264629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872d1316-e26b-4b6c-954e-47e554d5afca None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.277558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe71df2-30ba-462b-91c9-9036ca58813c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.278199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe71df2-30ba-462b-91c9-9036ca58813c admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.279218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe71df2-30ba-462b-91c9-9036ca58813c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.279500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe71df2-30ba-462b-91c9-9036ca58813c admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.280151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe71df2-30ba-462b-91c9-9036ca58813c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:02.280448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe71df2-30ba-462b-91c9-9036ca58813c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.283033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe71df2-30ba-462b-91c9-9036ca58813c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.305320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 982/1962] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:02.311831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe71df2-30ba-462b-91c9-9036ca58813c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.312190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28c8ddb-fc50-45a7-bb65-c3ab5866a6ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b8db4f8b50e496892d02241083f14f7, user_id=dd29e17afa534affa9f510a39b869f73, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.315823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28c8ddb-fc50-45a7-bb65-c3ab5866a6ad admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.320365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:02.325192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe71df2-30ba-462b-91c9-9036ca58813c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.348942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 981/1963] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:02 2022] PUT /identity/v3/projects/0b8db4f8b50e496892d02241083f14f7/users/dd29e17afa534affa9f510a39b869f73/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:02.360050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b98db2d-9144-48a2-bb1c-cfc633f8d28c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.360746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b98db2d-9144-48a2-bb1c-cfc633f8d28c admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.361180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b98db2d-9144-48a2-bb1c-cfc633f8d28c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.361539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b98db2d-9144-48a2-bb1c-cfc633f8d28c admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.362505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b98db2d-9144-48a2-bb1c-cfc633f8d28c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.365875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b98db2d-9144-48a2-bb1c-cfc633f8d28c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.368189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b468b8c-5254-4ff5-9b9c-88e38ba5eac2 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.368499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b468b8c-5254-4ff5-9b9c-88e38ba5eac2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.369172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b468b8c-5254-4ff5-9b9c-88e38ba5eac2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.388639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:02.416590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0b468b8c-5254-4ff5-9b9c-88e38ba5eac2 None None] MFA Rules not processed for user `dd29e17afa534affa9f510a39b869f73`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:02.417835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0b468b8c-5254-4ff5-9b9c-88e38ba5eac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.426238 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 983/1964] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:02 2022] POST /identity/v3/projects => generated 383 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:02.427268 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b98db2d-9144-48a2-bb1c-cfc633f8d28c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.592115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 982/1965] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:02 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:02.617594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b98db2d-9144-48a2-bb1c-cfc633f8d28c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.634843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b468b8c-5254-4ff5-9b9c-88e38ba5eac2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.636550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0b468b8c-5254-4ff5-9b9c-88e38ba5eac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.637376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53253a2-7980-49fe-8795-5834b800bef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.638039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53253a2-7980-49fe-8795-5834b800bef7 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.638355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53253a2-7980-49fe-8795-5834b800bef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.638629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53253a2-7980-49fe-8795-5834b800bef7 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.642205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53253a2-7980-49fe-8795-5834b800bef7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.642205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53253a2-7980-49fe-8795-5834b800bef7 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.654078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b468b8c-5254-4ff5-9b9c-88e38ba5eac2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.658734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320286b5-156b-4185-bac0-278037414453 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.659797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320286b5-156b-4185-bac0-278037414453 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.660074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320286b5-156b-4185-bac0-278037414453 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.660322 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320286b5-156b-4185-bac0-278037414453 admin admin] PATH_INFO: `/v3/users/d58bde7cde58465d893d26d99893bbe5` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.661581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320286b5-156b-4185-bac0-278037414453 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d58bde7cde58465d893d26d99893bbe5)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.663716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320286b5-156b-4185-bac0-278037414453 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.676783 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 984/1966] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:02 2022] POST /identity/v3/users => generated 292 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:02.677756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53253a2-7980-49fe-8795-5834b800bef7 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.679559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b53253a2-7980-49fe-8795-5834b800bef7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.737619 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-320286b5-156b-4185-bac0-278037414453 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd58bde7cde58465d893d26d99893bbe5'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.748026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-320286b5-156b-4185-bac0-278037414453 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd58bde7cde58465d893d26d99893bbe5'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.749133 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-320286b5-156b-4185-bac0-278037414453 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd58bde7cde58465d893d26d99893bbe5'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.750088 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-320286b5-156b-4185-bac0-278037414453 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd58bde7cde58465d893d26d99893bbe5'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.751083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53253a2-7980-49fe-8795-5834b800bef7 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.756879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5f1a82-c04d-4042-9ad6-8a004a18be7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.757524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5f1a82-c04d-4042-9ad6-8a004a18be7e service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.757811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5f1a82-c04d-4042-9ad6-8a004a18be7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.758081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5f1a82-c04d-4042-9ad6-8a004a18be7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.760191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-320286b5-156b-4185-bac0-278037414453 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd58bde7cde58465d893d26d99893bbe5'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.760349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2e5f1a82-c04d-4042-9ad6-8a004a18be7e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.772469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5f1a82-c04d-4042-9ad6-8a004a18be7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.774297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5f1a82-c04d-4042-9ad6-8a004a18be7e service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.787140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 983/1967] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:02 2022] DELETE /identity/v3/users/d58bde7cde58465d893d26d99893bbe5 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:02.787337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 985/1968] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:02 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:02.787452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320286b5-156b-4185-bac0-278037414453 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.787452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-320286b5-156b-4185-bac0-278037414453 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.788699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e59943-b0ec-4436-9051-e16fe6a441cc None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.789114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e59943-b0ec-4436-9051-e16fe6a441cc None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.789387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e59943-b0ec-4436-9051-e16fe6a441cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.813909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-17e59943-b0ec-4436-9051-e16fe6a441cc None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:02.815162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-17e59943-b0ec-4436-9051-e16fe6a441cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.873482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:02.879649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320286b5-156b-4185-bac0-278037414453 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.879841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 986/1969] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:02.880697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e59943-b0ec-4436-9051-e16fe6a441cc None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.882056 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-17e59943-b0ec-4436-9051-e16fe6a441cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.890055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e9ce8f-1fb5-4c76-9335-b15216dc5b5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.891176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e9ce8f-1fb5-4c76-9335-b15216dc5b5c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.891419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e9ce8f-1fb5-4c76-9335-b15216dc5b5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.891700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e9ce8f-1fb5-4c76-9335-b15216dc5b5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.892372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e9ce8f-1fb5-4c76-9335-b15216dc5b5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.894742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e9ce8f-1fb5-4c76-9335-b15216dc5b5c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.895820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e59943-b0ec-4436-9051-e16fe6a441cc None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.901338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.902278 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.902555 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.903432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] PATH_INFO: `/v3/users/dd29e17afa534affa9f510a39b869f73` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.904751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd29e17afa534affa9f510a39b869f73)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:02.906917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:02.908528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 984/1970] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:02 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:02.909429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e9ce8f-1fb5-4c76-9335-b15216dc5b5c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:02.911252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b0e9ce8f-1fb5-4c76-9335-b15216dc5b5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:02.925224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:02.964918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd29e17afa534affa9f510a39b869f73'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.965687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd29e17afa534affa9f510a39b869f73'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.975572 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd29e17afa534affa9f510a39b869f73'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.976241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd29e17afa534affa9f510a39b869f73'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.979535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e9ce8f-1fb5-4c76-9335-b15216dc5b5c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:02.984027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4644d151-cb24-4f2c-b445-793a9af6a3da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:02.984027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4644d151-cb24-4f2c-b445-793a9af6a3da service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:02.984273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4644d151-cb24-4f2c-b445-793a9af6a3da service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:02.986270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd29e17afa534affa9f510a39b869f73'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:02.993294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4644d151-cb24-4f2c-b445-793a9af6a3da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:02.996511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4644d151-cb24-4f2c-b445-793a9af6a3da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:03.008947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4644d151-cb24-4f2c-b445-793a9af6a3da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:03.023013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4644d151-cb24-4f2c-b445-793a9af6a3da service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:03.024012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 987/1971] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:02 2022] DELETE /identity/v3/users/dd29e17afa534affa9f510a39b869f73 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:03.027325 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:03.030536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:03.064586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 985/1972] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:03.067807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f367a89-0f85-4464-bff9-86efc8478b72 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:03.068785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f367a89-0f85-4464-bff9-86efc8478b72 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:03.068785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f367a89-0f85-4464-bff9-86efc8478b72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:03.128897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-6f367a89-0f85-4464-bff9-86efc8478b72 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:03.130814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6f367a89-0f85-4464-bff9-86efc8478b72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:03.160226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38510cf2-66d2-4c1c-8f5c-35652351e45a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:03.166920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2381df3b-0447-480d-8df4-35a30503fafd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:03.167604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2381df3b-0447-480d-8df4-35a30503fafd admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:03.167887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2381df3b-0447-480d-8df4-35a30503fafd admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:03.168434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2381df3b-0447-480d-8df4-35a30503fafd admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:03.169090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2381df3b-0447-480d-8df4-35a30503fafd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:03.170809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2381df3b-0447-480d-8df4-35a30503fafd admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:03.186987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 988/1973] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:03 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:03.191932 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2381df3b-0447-480d-8df4-35a30503fafd admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:03.205747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2381df3b-0447-480d-8df4-35a30503fafd admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:03.211030 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c54a35-bd77-42d4-bfe1-53d3eb5d912d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:03.211656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c54a35-bd77-42d4-bfe1-53d3eb5d912d admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:03.211942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c54a35-bd77-42d4-bfe1-53d3eb5d912d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:03.212207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c54a35-bd77-42d4-bfe1-53d3eb5d912d admin admin] PATH_INFO: `/v3/projects/b61f7b25402d402ba8c8967d0780dcc0/users/889b87a1b33647c198d0e95ac9fbd982/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:03.233109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c54a35-bd77-42d4-bfe1-53d3eb5d912d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b61f7b25402d402ba8c8967d0780dcc0, user_id=889b87a1b33647c198d0e95ac9fbd982, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:03.239395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c54a35-bd77-42d4-bfe1-53d3eb5d912d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:03.240393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 986/1974] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:03.254722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 989/1975] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:03 2022] PUT /identity/v3/projects/b61f7b25402d402ba8c8967d0780dcc0/users/889b87a1b33647c198d0e95ac9fbd982/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:03.263508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f367a89-0f85-4464-bff9-86efc8478b72 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:03.266336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6f367a89-0f85-4464-bff9-86efc8478b72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:03.268418 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b85e351-f1d5-47b3-8c8c-d0612283821e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:03.268703 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b85e351-f1d5-47b3-8c8c-d0612283821e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:03.268972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b85e351-f1d5-47b3-8c8c-d0612283821e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:03.314941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3b85e351-f1d5-47b3-8c8c-d0612283821e None None] MFA Rules not processed for user `889b87a1b33647c198d0e95ac9fbd982`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:03.316311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3b85e351-f1d5-47b3-8c8c-d0612283821e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:03.391607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f367a89-0f85-4464-bff9-86efc8478b72 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:03.416240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:03.462265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 990/1976] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:03 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:03.473227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b85e351-f1d5-47b3-8c8c-d0612283821e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:03.476130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3b85e351-f1d5-47b3-8c8c-d0612283821e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:03.514130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9855eb8-661e-49f4-a810-4b192fff8e60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:03.514130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9855eb8-661e-49f4-a810-4b192fff8e60 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:03.514130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9855eb8-661e-49f4-a810-4b192fff8e60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:03.514130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9855eb8-661e-49f4-a810-4b192fff8e60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:03.521130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:03.612428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b85e351-f1d5-47b3-8c8c-d0612283821e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:03.620855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3218b1f0-de7c-4660-91ba-1b02625b47b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:03.621490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3218b1f0-de7c-4660-91ba-1b02625b47b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:03.621762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3218b1f0-de7c-4660-91ba-1b02625b47b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:03.622023 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3218b1f0-de7c-4660-91ba-1b02625b47b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:03.646855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3218b1f0-de7c-4660-91ba-1b02625b47b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:03.646855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3218b1f0-de7c-4660-91ba-1b02625b47b0 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:03.660157 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9855eb8-661e-49f4-a810-4b192fff8e60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:03.667174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9855eb8-661e-49f4-a810-4b192fff8e60 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:03.689522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 991/1977] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:03 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:03.697124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 987/1978] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:03.722780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3218b1f0-de7c-4660-91ba-1b02625b47b0 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:03.737121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3218b1f0-de7c-4660-91ba-1b02625b47b0 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:03.749149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:03.749995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:03.750533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:03.750986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] PATH_INFO: `/v3/users/889b87a1b33647c198d0e95ac9fbd982` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:03.754398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=889b87a1b33647c198d0e95ac9fbd982)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:03.757164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:03.852857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '889b87a1b33647c198d0e95ac9fbd982'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:03.854839 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '889b87a1b33647c198d0e95ac9fbd982'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:03.868885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '889b87a1b33647c198d0e95ac9fbd982'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:03.869951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '889b87a1b33647c198d0e95ac9fbd982'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:03.881783 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '889b87a1b33647c198d0e95ac9fbd982'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:03.916670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 992/1979] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:03 2022] DELETE /identity/v3/users/889b87a1b33647c198d0e95ac9fbd982 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:03.921226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083e652a-c710-434d-a790-e230fa5a60f1 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:03.921500 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083e652a-c710-434d-a790-e230fa5a60f1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:03.921865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083e652a-c710-434d-a790-e230fa5a60f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:03.968465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-083e652a-c710-434d-a790-e230fa5a60f1 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:03.969760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-083e652a-c710-434d-a790-e230fa5a60f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:04.039881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 988/1980] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:04.060070 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:04.061740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:04.095423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083e652a-c710-434d-a790-e230fa5a60f1 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:04.097165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-083e652a-c710-434d-a790-e230fa5a60f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:04.118245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083e652a-c710-434d-a790-e230fa5a60f1 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:04.127107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:04.127964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:04.128107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:04.128361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] PATH_INFO: `/v3/projects/0c606390d99e45faa1970a54148a51f2` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:04.129961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] RBAC: Authorizing `identity:delete_project(project_id=0c606390d99e45faa1970a54148a51f2)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:04.163679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b868743-f2e8-400a-b165-69989e94bfe4 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:04.182723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:04.189286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e546d29-1965-4882-9f13-4ae039dfec9c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:04.189956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e546d29-1965-4882-9f13-4ae039dfec9c service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:04.190438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e546d29-1965-4882-9f13-4ae039dfec9c service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:04.190707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e546d29-1965-4882-9f13-4ae039dfec9c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:04.217696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e546d29-1965-4882-9f13-4ae039dfec9c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:04.219425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e546d29-1965-4882-9f13-4ae039dfec9c service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:04.231529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 993/1981] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:04.265065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0c606390d99e45faa1970a54148a51f2', 'request_id': 'req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.272836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0c606390d99e45faa1970a54148a51f2', 'request_id': 'req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.282721 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0c606390d99e45faa1970a54148a51f2', 'request_id': 'req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.297160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] The token cache is being invalidate because project 0c606390d99e45faa1970a54148a51f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:04.297411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.298794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.301162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 989/1982] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:04 2022] DELETE /identity/v3/projects/0c606390d99e45faa1970a54148a51f2 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:04.305443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e546d29-1965-4882-9f13-4ae039dfec9c service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:04.317739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e546d29-1965-4882-9f13-4ae039dfec9c service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:04.328043 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:04.387653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:04.388553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:04.388960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:04.389336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] PATH_INFO: `/v3/users/b2f3818017324197b9b7143fb2fa8d25` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:04.391140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2f3818017324197b9b7143fb2fa8d25)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:04.392942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:04.464120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2f3818017324197b9b7143fb2fa8d25'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.465349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2f3818017324197b9b7143fb2fa8d25'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.474672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2f3818017324197b9b7143fb2fa8d25'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.475624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2f3818017324197b9b7143fb2fa8d25'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.484884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2f3818017324197b9b7143fb2fa8d25'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.515063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 994/1983] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:04 2022] DELETE /identity/v3/users/b2f3818017324197b9b7143fb2fa8d25 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:04.575389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:04.577618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:04.593461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19a5a5d-048f-4048-bd28-ba5b1dc5185b admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:04.643331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:04.724279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:04.725169 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:04.725543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:04.726203 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] PATH_INFO: `/v3/projects/0b8db4f8b50e496892d02241083f14f7` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:04.729342 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b8db4f8b50e496892d02241083f14f7)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:04.732653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:04.781296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:04.782844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:04.797217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c9ade7-27a5-481f-a482-0167f8266e85 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:04.802495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:04.803876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:04.804469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:04.804919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] PATH_INFO: `/v3/projects/b61f7b25402d402ba8c8967d0780dcc0` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:04.806873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b61f7b25402d402ba8c8967d0780dcc0)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:04.809255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:04.847453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b8db4f8b50e496892d02241083f14f7', 'request_id': 'req-76e638aa-789b-45c1-aa42-8ce80cf1cefa'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.860648 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b8db4f8b50e496892d02241083f14f7', 'request_id': 'req-76e638aa-789b-45c1-aa42-8ce80cf1cefa'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.888104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b8db4f8b50e496892d02241083f14f7', 'request_id': 'req-76e638aa-789b-45c1-aa42-8ce80cf1cefa'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.918230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b61f7b25402d402ba8c8967d0780dcc0', 'request_id': 'req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.927776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b61f7b25402d402ba8c8967d0780dcc0', 'request_id': 'req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.933317 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] The token cache is being invalidate because project 0b8db4f8b50e496892d02241083f14f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:04.933611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.935001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-76e638aa-789b-45c1-aa42-8ce80cf1cefa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.939103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 990/1984] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:04 2022] DELETE /identity/v3/projects/0b8db4f8b50e496892d02241083f14f7 => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:04.941987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b61f7b25402d402ba8c8967d0780dcc0', 'request_id': 'req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.944113 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9936f6-9455-4a59-aae3-112ff6b0f525 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:04.944881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9936f6-9455-4a59-aae3-112ff6b0f525 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:04.945166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9936f6-9455-4a59-aae3-112ff6b0f525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:04.972567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7f9936f6-9455-4a59-aae3-112ff6b0f525 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:04.974044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7f9936f6-9455-4a59-aae3-112ff6b0f525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:04.975844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] The token cache is being invalidate because project b61f7b25402d402ba8c8967d0780dcc0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:04.976194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.978974 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2f86fb42-69f8-4ddb-bd6a-9bd5f420935e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:04.982461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 995/1985] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:04 2022] DELETE /identity/v3/projects/b61f7b25402d402ba8c8967d0780dcc0 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:04.988962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d688c1-3a84-41d7-a263-cfc1cdbc27ee None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:04.989565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d688c1-3a84-41d7-a263-cfc1cdbc27ee None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:04.990130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d688c1-3a84-41d7-a263-cfc1cdbc27ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.050547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d3d688c1-3a84-41d7-a263-cfc1cdbc27ee None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:05.051871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d3d688c1-3a84-41d7-a263-cfc1cdbc27ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:05.066690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 991/1986] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:05.074241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c915fd1e-5a4d-4df8-bc94-3befe1c0a522 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.074527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c915fd1e-5a4d-4df8-bc94-3befe1c0a522 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.074855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c915fd1e-5a4d-4df8-bc94-3befe1c0a522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.076600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 996/1987] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:05.084061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9d3950-e7f9-466f-a4a7-f2d3fd93e5fa None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.084613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9d3950-e7f9-466f-a4a7-f2d3fd93e5fa None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.084977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9d3950-e7f9-466f-a4a7-f2d3fd93e5fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.105023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c915fd1e-5a4d-4df8-bc94-3befe1c0a522 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:05.106548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c915fd1e-5a4d-4df8-bc94-3befe1c0a522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:05.122595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 992/1988] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:05.123794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-ab9d3950-e7f9-466f-a4a7-f2d3fd93e5fa None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:05.124717 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c915fd1e-5a4d-4df8-bc94-3befe1c0a522 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.125047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ab9d3950-e7f9-466f-a4a7-f2d3fd93e5fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:05.141543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 997/1989] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:05.143515 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c915fd1e-5a4d-4df8-bc94-3befe1c0a522 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.146510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9d3950-e7f9-466f-a4a7-f2d3fd93e5fa None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.154431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4c49f-7745-451c-8ed3-2bd071d87a8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.155729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:05.157601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4c49f-7745-451c-8ed3-2bd071d87a8f admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.157601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4c49f-7745-451c-8ed3-2bd071d87a8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.158509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4c49f-7745-451c-8ed3-2bd071d87a8f admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.159244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf4c49f-7745-451c-8ed3-2bd071d87a8f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:05.159612 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf4c49f-7745-451c-8ed3-2bd071d87a8f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.161282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf4c49f-7745-451c-8ed3-2bd071d87a8f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.163618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9d3950-e7f9-466f-a4a7-f2d3fd93e5fa None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.170797 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1083682-761a-4928-a020-e275d674674b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.171641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1083682-761a-4928-a020-e275d674674b admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.172587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1083682-761a-4928-a020-e275d674674b admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.173630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1083682-761a-4928-a020-e275d674674b admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.174655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1083682-761a-4928-a020-e275d674674b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:05.174773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1083682-761a-4928-a020-e275d674674b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.177448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1083682-761a-4928-a020-e275d674674b admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.185994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 993/1990] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:05.187028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4c49f-7745-451c-8ed3-2bd071d87a8f admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.194803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 998/1991] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:05.205932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4c49f-7745-451c-8ed3-2bd071d87a8f admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.212420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1083682-761a-4928-a020-e275d674674b admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.213087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8b32da-36ed-473f-a8a3-8e019469bf16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.214091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8b32da-36ed-473f-a8a3-8e019469bf16 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.214544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8b32da-36ed-473f-a8a3-8e019469bf16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.214973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8b32da-36ed-473f-a8a3-8e019469bf16 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.215810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8b32da-36ed-473f-a8a3-8e019469bf16 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.218091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8b32da-36ed-473f-a8a3-8e019469bf16 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.272355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1083682-761a-4928-a020-e275d674674b admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.296528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 994/1992] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:05 2022] POST /identity/v3/projects => generated 375 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:05.297605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8b32da-36ed-473f-a8a3-8e019469bf16 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.318871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101b0f48-a3a8-490a-8deb-5331b4173f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.318871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101b0f48-a3a8-490a-8deb-5331b4173f6c admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.318871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101b0f48-a3a8-490a-8deb-5331b4173f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.318871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101b0f48-a3a8-490a-8deb-5331b4173f6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.319669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101b0f48-a3a8-490a-8deb-5331b4173f6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.319754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-df8b32da-36ed-473f-a8a3-8e019469bf16 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:05.319840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101b0f48-a3a8-490a-8deb-5331b4173f6c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.355172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:05.364477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 999/1993] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:05 2022] POST /identity/v3/projects => generated 375 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:05.365651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101b0f48-a3a8-490a-8deb-5331b4173f6c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.477734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8b32da-36ed-473f-a8a3-8e019469bf16 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.507857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101b0f48-a3a8-490a-8deb-5331b4173f6c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.515067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f37c3fd-754e-4d53-93e5-98d1e6edbdbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.516323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f37c3fd-754e-4d53-93e5-98d1e6edbdbe admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.516624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f37c3fd-754e-4d53-93e5-98d1e6edbdbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.517037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f37c3fd-754e-4d53-93e5-98d1e6edbdbe admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.517943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f37c3fd-754e-4d53-93e5-98d1e6edbdbe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.520137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f37c3fd-754e-4d53-93e5-98d1e6edbdbe admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.560757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.561421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.561712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.562352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] PATH_INFO: `/v3/projects/b8e44671811d4b228bb4bd334df5cd43` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.563466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8e44671811d4b228bb4bd334df5cd43)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.563996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1000/1994] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:05 2022] POST /identity/v3/users => generated 288 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:05.565037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f37c3fd-754e-4d53-93e5-98d1e6edbdbe admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.566989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.584467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f37c3fd-754e-4d53-93e5-98d1e6edbdbe admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.592235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e85f92-1d49-4e5a-a8a4-e9f55ee50483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.593253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e85f92-1d49-4e5a-a8a4-e9f55ee50483 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.593705 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e85f92-1d49-4e5a-a8a4-e9f55ee50483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.594158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e85f92-1d49-4e5a-a8a4-e9f55ee50483 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.595076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e85f92-1d49-4e5a-a8a4-e9f55ee50483 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.599218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e85f92-1d49-4e5a-a8a4-e9f55ee50483 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.638502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1001/1995] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:05 2022] POST /identity/v3/users => generated 288 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:05.640147 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e85f92-1d49-4e5a-a8a4-e9f55ee50483 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.645611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8e44671811d4b228bb4bd334df5cd43', 'request_id': 'req-a8732f99-3bff-44ea-9d77-258dd39c72e6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:05.655448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e85f92-1d49-4e5a-a8a4-e9f55ee50483 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.656318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8e44671811d4b228bb4bd334df5cd43', 'request_id': 'req-a8732f99-3bff-44ea-9d77-258dd39c72e6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:05.672905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8e44671811d4b228bb4bd334df5cd43', 'request_id': 'req-a8732f99-3bff-44ea-9d77-258dd39c72e6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:05.694796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] The token cache is being invalidate because project b8e44671811d4b228bb4bd334df5cd43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:05.695360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:05.696800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:05.698617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 995/1996] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:05 2022] DELETE /identity/v3/projects/b8e44671811d4b228bb4bd334df5cd43 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:05.701933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.701933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:05.747519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf525fe-269f-46e6-94ba-6ade3564d9ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.748696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf525fe-269f-46e6-94ba-6ade3564d9ca admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.749082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf525fe-269f-46e6-94ba-6ade3564d9ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.749460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf525fe-269f-46e6-94ba-6ade3564d9ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.750694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf525fe-269f-46e6-94ba-6ade3564d9ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.762424 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf525fe-269f-46e6-94ba-6ade3564d9ca admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.768997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1002/1997] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:05 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:05.770769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5ea46c-808a-404f-866f-3565f005791b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.771428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5ea46c-808a-404f-866f-3565f005791b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.771899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5ea46c-808a-404f-866f-3565f005791b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.781752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:05.794231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-9f5ea46c-808a-404f-866f-3565f005791b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:05.795732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9f5ea46c-808a-404f-866f-3565f005791b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:05.797864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8732f99-3bff-44ea-9d77-258dd39c72e6 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.803702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dda519d-5782-4b32-bf77-127929d4462c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.811113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1003/1998] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:05.812125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda519d-5782-4b32-bf77-127929d4462c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.812125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda519d-5782-4b32-bf77-127929d4462c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.812308 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5ea46c-808a-404f-866f-3565f005791b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.812838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dda519d-5782-4b32-bf77-127929d4462c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.812838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dda519d-5782-4b32-bf77-127929d4462c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.813007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9f5ea46c-808a-404f-866f-3565f005791b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:05.813085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dda519d-5782-4b32-bf77-127929d4462c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.819006 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5ea46c-808a-404f-866f-3565f005791b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.819454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 996/1999] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:05 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:05.820647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e6ac79-9051-4ba2-a8b4-d6a643eb1591 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.820963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e6ac79-9051-4ba2-a8b4-d6a643eb1591 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.821237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e6ac79-9051-4ba2-a8b4-d6a643eb1591 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.824088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc73bfa5-a8e9-4737-a78b-b7fc16c6b11b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.825230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc73bfa5-a8e9-4737-a78b-b7fc16c6b11b admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.825230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc73bfa5-a8e9-4737-a78b-b7fc16c6b11b admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.825230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc73bfa5-a8e9-4737-a78b-b7fc16c6b11b admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.825869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc73bfa5-a8e9-4737-a78b-b7fc16c6b11b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.827518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc73bfa5-a8e9-4737-a78b-b7fc16c6b11b admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.842545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1004/2000] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:05 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:05.842545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc73bfa5-a8e9-4737-a78b-b7fc16c6b11b admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.844061 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-41e6ac79-9051-4ba2-a8b4-d6a643eb1591 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:05.845273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-41e6ac79-9051-4ba2-a8b4-d6a643eb1591 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:05.853000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc73bfa5-a8e9-4737-a78b-b7fc16c6b11b admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.853839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 997/2001] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:05.858393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e6ac79-9051-4ba2-a8b4-d6a643eb1591 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.859026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3733381f-a210-4eb4-a947-1b5b152ce21d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.859937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3733381f-a210-4eb4-a947-1b5b152ce21d admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.860230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3733381f-a210-4eb4-a947-1b5b152ce21d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.860612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3733381f-a210-4eb4-a947-1b5b152ce21d admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.861394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3733381f-a210-4eb4-a947-1b5b152ce21d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.863098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3733381f-a210-4eb4-a947-1b5b152ce21d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.868238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e6ac79-9051-4ba2-a8b4-d6a643eb1591 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.874633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d43cf58-fc04-4857-bac0-00535abe738e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.874840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1005/2002] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:05 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:05.875380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d43cf58-fc04-4857-bac0-00535abe738e admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.875657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d43cf58-fc04-4857-bac0-00535abe738e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.875902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3733381f-a210-4eb4-a947-1b5b152ce21d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.876034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d43cf58-fc04-4857-bac0-00535abe738e admin admin] PATH_INFO: `/v3/projects/d78f523a266a403bb42f78dcfbaa68ed/users/55af8787cc8c4e8bb4b835d6a483d7c1/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.886541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3733381f-a210-4eb4-a947-1b5b152ce21d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:05.891542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7072f5-4594-4a93-a8f1-c212ef953bab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:05.892232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7072f5-4594-4a93-a8f1-c212ef953bab admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:05.892600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7072f5-4594-4a93-a8f1-c212ef953bab admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:05.892940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7072f5-4594-4a93-a8f1-c212ef953bab admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:05.893327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d43cf58-fc04-4857-bac0-00535abe738e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d78f523a266a403bb42f78dcfbaa68ed, user_id=55af8787cc8c4e8bb4b835d6a483d7c1, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.893891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7072f5-4594-4a93-a8f1-c212ef953bab admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:05.894190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7072f5-4594-4a93-a8f1-c212ef953bab admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:05.895967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7072f5-4594-4a93-a8f1-c212ef953bab admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.896882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d43cf58-fc04-4857-bac0-00535abe738e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:05.897640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:05.912086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1006/2003] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:05.913269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7072f5-4594-4a93-a8f1-c212ef953bab admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.926039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 998/2004] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:05 2022] PUT /identity/v3/projects/d78f523a266a403bb42f78dcfbaa68ed/users/55af8787cc8c4e8bb4b835d6a483d7c1/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:05.926039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d43cf58-fc04-4857-bac0-00535abe738e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:05.935938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7072f5-4594-4a93-a8f1-c212ef953bab admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:06.031695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d43cf58-fc04-4857-bac0-00535abe738e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:06.051884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922ac5d5-bfa9-49aa-9e12-fdedaf36f08b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:06.054825 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922ac5d5-bfa9-49aa-9e12-fdedaf36f08b admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.055059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922ac5d5-bfa9-49aa-9e12-fdedaf36f08b admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.055330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922ac5d5-bfa9-49aa-9e12-fdedaf36f08b admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.056078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922ac5d5-bfa9-49aa-9e12-fdedaf36f08b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:06.058035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922ac5d5-bfa9-49aa-9e12-fdedaf36f08b admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:06.071047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0fdc70-c5fc-419c-b901-179db2e9c303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:06.072208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0fdc70-c5fc-419c-b901-179db2e9c303 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.072570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0fdc70-c5fc-419c-b901-179db2e9c303 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.072973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0fdc70-c5fc-419c-b901-179db2e9c303 admin admin] PATH_INFO: `/v3/projects/9f2a675882b549238f66b56ab1aa21af/users/2045e938699844569c177d0f37229fe3/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.094310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0fdc70-c5fc-419c-b901-179db2e9c303 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f2a675882b549238f66b56ab1aa21af, user_id=2045e938699844569c177d0f37229fe3, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:06.097621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0fdc70-c5fc-419c-b901-179db2e9c303 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:06.102366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 999/2005] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:05 2022] POST /identity/v3/projects => generated 387 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:06.102366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c17af68-f0fc-4d9a-9a48-d43c5a261945 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.102366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c17af68-f0fc-4d9a-9a48-d43c5a261945 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.102366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c17af68-f0fc-4d9a-9a48-d43c5a261945 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.115046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1007/2006] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:05 2022] PUT /identity/v3/projects/9f2a675882b549238f66b56ab1aa21af/users/2045e938699844569c177d0f37229fe3/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:06.116814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0fdc70-c5fc-419c-b901-179db2e9c303 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:06.142162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4c17af68-f0fc-4d9a-9a48-d43c5a261945 None None] MFA Rules not processed for user `55af8787cc8c4e8bb4b835d6a483d7c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:06.143398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4c17af68-f0fc-4d9a-9a48-d43c5a261945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:06.201864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0fdc70-c5fc-419c-b901-179db2e9c303 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:06.208671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94a7f9b-a6d9-4300-aa21-f9c0a0534d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:06.209594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94a7f9b-a6d9-4300-aa21-f9c0a0534d4a admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.210022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94a7f9b-a6d9-4300-aa21-f9c0a0534d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.210328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94a7f9b-a6d9-4300-aa21-f9c0a0534d4a admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.210960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94a7f9b-a6d9-4300-aa21-f9c0a0534d4a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:06.213336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1000/2007] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:06.213462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94a7f9b-a6d9-4300-aa21-f9c0a0534d4a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:06.214748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e125ac-409f-401d-8155-45b093e327d7 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.215106 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e125ac-409f-401d-8155-45b093e327d7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.215465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e125ac-409f-401d-8155-45b093e327d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.242229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1008/2008] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:06 2022] POST /identity/v3/users => generated 294 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:06.242229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94a7f9b-a6d9-4300-aa21-f9c0a0534d4a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:06.243741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a94a7f9b-a6d9-4300-aa21-f9c0a0534d4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:06.247055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-52e125ac-409f-401d-8155-45b093e327d7 None None] MFA Rules not processed for user `2045e938699844569c177d0f37229fe3`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:06.248255 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-52e125ac-409f-401d-8155-45b093e327d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:06.269444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:06.311754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94a7f9b-a6d9-4300-aa21-f9c0a0534d4a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:06.315371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b06152c-e838-4532-bd23-fdd007471a26 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:06.315975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b06152c-e838-4532-bd23-fdd007471a26 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.316252 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b06152c-e838-4532-bd23-fdd007471a26 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.316520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b06152c-e838-4532-bd23-fdd007471a26 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.321236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1001/2009] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:06.322131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e125ac-409f-401d-8155-45b093e327d7 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:06.327885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b06152c-e838-4532-bd23-fdd007471a26 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:06.329480 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b06152c-e838-4532-bd23-fdd007471a26 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:06.335448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e125ac-409f-401d-8155-45b093e327d7 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:06.341494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1009/2010] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:06 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:06.342424 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b06152c-e838-4532-bd23-fdd007471a26 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:06.343766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0b06152c-e838-4532-bd23-fdd007471a26 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:06.344207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3bcce-6b3a-4875-b15d-9d5011a041c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:06.345541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3bcce-6b3a-4875-b15d-9d5011a041c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.345928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3bcce-6b3a-4875-b15d-9d5011a041c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.346859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3bcce-6b3a-4875-b15d-9d5011a041c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.348446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb3bcce-6b3a-4875-b15d-9d5011a041c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:06.351457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb3bcce-6b3a-4875-b15d-9d5011a041c6 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:06.358667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b06152c-e838-4532-bd23-fdd007471a26 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:06.363287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d4cb18-670a-4099-8383-1c8cb6b1d6c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:06.363953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4cb18-670a-4099-8383-1c8cb6b1d6c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.364232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4cb18-670a-4099-8383-1c8cb6b1d6c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.364501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4cb18-670a-4099-8383-1c8cb6b1d6c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.369587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1002/2011] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:06 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:06.370722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3bcce-6b3a-4875-b15d-9d5011a041c6 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:06.374492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:06.377086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d4cb18-670a-4099-8383-1c8cb6b1d6c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:06.379318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d4cb18-670a-4099-8383-1c8cb6b1d6c3 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:06.384775 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3bcce-6b3a-4875-b15d-9d5011a041c6 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:06.391199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503817ce-af19-48a9-8ff9-8f44da2176ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:06.392074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503817ce-af19-48a9-8ff9-8f44da2176ee admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.392353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503817ce-af19-48a9-8ff9-8f44da2176ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.392813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503817ce-af19-48a9-8ff9-8f44da2176ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.393706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1010/2012] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:06 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:06.393822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503817ce-af19-48a9-8ff9-8f44da2176ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:06.395899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503817ce-af19-48a9-8ff9-8f44da2176ee admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:06.407505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1003/2013] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:06 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:06.411569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d4cb18-670a-4099-8383-1c8cb6b1d6c3 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:06.424404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d4cb18-670a-4099-8383-1c8cb6b1d6c3 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:06.430490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac44e049-5fdb-463a-8777-bd33f4e98121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:06.431206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac44e049-5fdb-463a-8777-bd33f4e98121 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.431481 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac44e049-5fdb-463a-8777-bd33f4e98121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.431761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac44e049-5fdb-463a-8777-bd33f4e98121 admin admin] PATH_INFO: `/v3/projects/fd5ad15b266e4ea58300b5dff91e037f/users/85e97eae4f084f6cb298dd1a98bdbfa2/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.449953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac44e049-5fdb-463a-8777-bd33f4e98121 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd5ad15b266e4ea58300b5dff91e037f, user_id=85e97eae4f084f6cb298dd1a98bdbfa2, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:06.450204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac44e049-5fdb-463a-8777-bd33f4e98121 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:06.480262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1011/2014] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:06 2022] PUT /identity/v3/projects/fd5ad15b266e4ea58300b5dff91e037f/users/85e97eae4f084f6cb298dd1a98bdbfa2/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:06.504187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad9a2a1-5317-46e4-8d50-9f962c5b22c7 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.506449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad9a2a1-5317-46e4-8d50-9f962c5b22c7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.507264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad9a2a1-5317-46e4-8d50-9f962c5b22c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.572122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-fad9a2a1-5317-46e4-8d50-9f962c5b22c7 None None] MFA Rules not processed for user `85e97eae4f084f6cb298dd1a98bdbfa2`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:06.574718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-fad9a2a1-5317-46e4-8d50-9f962c5b22c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:06.710611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1004/2015] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:06 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:06.712824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac44e049-5fdb-463a-8777-bd33f4e98121 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:06.871457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac44e049-5fdb-463a-8777-bd33f4e98121 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:06.884923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6623265-1c62-4b11-b034-fce847483d48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:06.885857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6623265-1c62-4b11-b034-fce847483d48 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:06.886289 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6623265-1c62-4b11-b034-fce847483d48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:06.886656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6623265-1c62-4b11-b034-fce847483d48 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:06.887392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6623265-1c62-4b11-b034-fce847483d48 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:06.903879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6623265-1c62-4b11-b034-fce847483d48 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:06.962903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1012/2016] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:06 2022] POST /identity/v3/projects => generated 387 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:06.967046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad9a2a1-5317-46e4-8d50-9f962c5b22c7 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:06.982055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad9a2a1-5317-46e4-8d50-9f962c5b22c7 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:07.063588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaf3145-bd45-456d-b0d4-7bf2d1908ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:07.064373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaf3145-bd45-456d-b0d4-7bf2d1908ac5 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.064836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaf3145-bd45-456d-b0d4-7bf2d1908ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.065844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaf3145-bd45-456d-b0d4-7bf2d1908ac5 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.066694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfaf3145-bd45-456d-b0d4-7bf2d1908ac5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:07.070052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfaf3145-bd45-456d-b0d4-7bf2d1908ac5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:07.100432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1005/2017] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:06 2022] POST /identity/v3/users => generated 294 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:07.106356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6623265-1c62-4b11-b034-fce847483d48 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:07.118338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6623265-1c62-4b11-b034-fce847483d48 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:07.127306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ca725-d2e8-4c2f-a504-2303ac3f6adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:07.128391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ca725-d2e8-4c2f-a504-2303ac3f6adc admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.128678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ca725-d2e8-4c2f-a504-2303ac3f6adc admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.128953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ca725-d2e8-4c2f-a504-2303ac3f6adc admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.129609 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ca725-d2e8-4c2f-a504-2303ac3f6adc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:07.131291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ca725-d2e8-4c2f-a504-2303ac3f6adc admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:07.146226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1013/2018] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:07 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:07.150320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaf3145-bd45-456d-b0d4-7bf2d1908ac5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:07.162824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaf3145-bd45-456d-b0d4-7bf2d1908ac5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:07.168356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84a6453-eca3-4d69-beda-ed0a5dffa59d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:07.169089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84a6453-eca3-4d69-beda-ed0a5dffa59d admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.169450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84a6453-eca3-4d69-beda-ed0a5dffa59d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.169831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84a6453-eca3-4d69-beda-ed0a5dffa59d admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.170535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84a6453-eca3-4d69-beda-ed0a5dffa59d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:07.172343 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84a6453-eca3-4d69-beda-ed0a5dffa59d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:07.184682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1006/2019] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:07 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:31:07.190430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ca725-d2e8-4c2f-a504-2303ac3f6adc admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:07.200928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ca725-d2e8-4c2f-a504-2303ac3f6adc admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:07.205510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3f0ffc-72aa-46d6-a117-36cf7bcc499d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:07.206186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3f0ffc-72aa-46d6-a117-36cf7bcc499d admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.206464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3f0ffc-72aa-46d6-a117-36cf7bcc499d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.206732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3f0ffc-72aa-46d6-a117-36cf7bcc499d admin admin] PATH_INFO: `/v3/projects/062dc9d61b5b4bab93d847b7b4011cca/users/c19b90915e8c46998afaecfc32d9f9ec/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.222090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3f0ffc-72aa-46d6-a117-36cf7bcc499d admin admin] RBAC: Authorizing `identity:create_grant(project_id=062dc9d61b5b4bab93d847b7b4011cca, user_id=c19b90915e8c46998afaecfc32d9f9ec, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:07.224801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3f0ffc-72aa-46d6-a117-36cf7bcc499d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:07.241691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1014/2020] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:07 2022] PUT /identity/v3/projects/062dc9d61b5b4bab93d847b7b4011cca/users/c19b90915e8c46998afaecfc32d9f9ec/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:07.252141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699d98ef-ec88-4fa6-bd3e-c7d1d49da4f1 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.252435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699d98ef-ec88-4fa6-bd3e-c7d1d49da4f1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.252700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699d98ef-ec88-4fa6-bd3e-c7d1d49da4f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.266689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:07.287334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-699d98ef-ec88-4fa6-bd3e-c7d1d49da4f1 None None] MFA Rules not processed for user `c19b90915e8c46998afaecfc32d9f9ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:07.289628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-699d98ef-ec88-4fa6-bd3e-c7d1d49da4f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:07.428233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1007/2021] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:07 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:07.433153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3f0ffc-72aa-46d6-a117-36cf7bcc499d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:07.499037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3f0ffc-72aa-46d6-a117-36cf7bcc499d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:07.503912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140574b9-b82b-49b0-b124-bd5689f897b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:07.504533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140574b9-b82b-49b0-b124-bd5689f897b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.504809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140574b9-b82b-49b0-b124-bd5689f897b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.505077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140574b9-b82b-49b0-b124-bd5689f897b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.505699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140574b9-b82b-49b0-b124-bd5689f897b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:07.507646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140574b9-b82b-49b0-b124-bd5689f897b9 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:07.536701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1015/2022] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:07 2022] POST /identity/v3/projects => generated 387 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:07.541116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699d98ef-ec88-4fa6-bd3e-c7d1d49da4f1 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:07.551981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699d98ef-ec88-4fa6-bd3e-c7d1d49da4f1 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:07.615128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c3b328-3206-41d2-bdea-55bae7077eb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:07.615847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3b328-3206-41d2-bdea-55bae7077eb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.616115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3b328-3206-41d2-bdea-55bae7077eb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.616391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3b328-3206-41d2-bdea-55bae7077eb4 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.617038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c3b328-3206-41d2-bdea-55bae7077eb4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:07.619083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c3b328-3206-41d2-bdea-55bae7077eb4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:07.644926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1008/2023] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:07 2022] POST /identity/v3/users => generated 300 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:07.649737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140574b9-b82b-49b0-b124-bd5689f897b9 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:07.662511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140574b9-b82b-49b0-b124-bd5689f897b9 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:07.667847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6360ba-03d5-4bd4-a912-73ce41f5c04f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:07.668492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6360ba-03d5-4bd4-a912-73ce41f5c04f admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.668769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6360ba-03d5-4bd4-a912-73ce41f5c04f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.669071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6360ba-03d5-4bd4-a912-73ce41f5c04f admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.669714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6360ba-03d5-4bd4-a912-73ce41f5c04f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:07.671440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6360ba-03d5-4bd4-a912-73ce41f5c04f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:07.672727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:07.688709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1016/2024] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:07 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:07.694425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c3b328-3206-41d2-bdea-55bae7077eb4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:07.705803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c3b328-3206-41d2-bdea-55bae7077eb4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:07.711434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cf885f-cc3c-461b-8770-92d78c1d202d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:07.712189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cf885f-cc3c-461b-8770-92d78c1d202d admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.712482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cf885f-cc3c-461b-8770-92d78c1d202d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.712753 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cf885f-cc3c-461b-8770-92d78c1d202d admin admin] PATH_INFO: `/v3/projects/56741605cfd34e87878dca1ce623da03/users/b7266c6bbe6e426d89a12a0f5d1b6752/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.730861 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cf885f-cc3c-461b-8770-92d78c1d202d admin admin] RBAC: Authorizing `identity:create_grant(project_id=56741605cfd34e87878dca1ce623da03, user_id=b7266c6bbe6e426d89a12a0f5d1b6752, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:07.733657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cf885f-cc3c-461b-8770-92d78c1d202d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:07.748547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1009/2025] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:07 2022] PUT /identity/v3/projects/56741605cfd34e87878dca1ce623da03/users/b7266c6bbe6e426d89a12a0f5d1b6752/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:07.752873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6360ba-03d5-4bd4-a912-73ce41f5c04f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:07.762988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6360ba-03d5-4bd4-a912-73ce41f5c04f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:07.834670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57613674-8b1c-489e-ae92-223ba28b7754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:07.835493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57613674-8b1c-489e-ae92-223ba28b7754 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.835872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57613674-8b1c-489e-ae92-223ba28b7754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.836243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57613674-8b1c-489e-ae92-223ba28b7754 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.837158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57613674-8b1c-489e-ae92-223ba28b7754 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:07.839250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57613674-8b1c-489e-ae92-223ba28b7754 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:07.862067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1017/2026] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:07 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:07.866348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cf885f-cc3c-461b-8770-92d78c1d202d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:07.885887 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cf885f-cc3c-461b-8770-92d78c1d202d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:07.900462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e4a6f6-2557-44b4-8a16-38304d01c1d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:07.901166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e4a6f6-2557-44b4-8a16-38304d01c1d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.901543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e4a6f6-2557-44b4-8a16-38304d01c1d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.901711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e4a6f6-2557-44b4-8a16-38304d01c1d9 admin admin] PATH_INFO: `/v3/projects/56741605cfd34e87878dca1ce623da03/users/b7266c6bbe6e426d89a12a0f5d1b6752/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.909460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e4a6f6-2557-44b4-8a16-38304d01c1d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56741605cfd34e87878dca1ce623da03, user_id=b7266c6bbe6e426d89a12a0f5d1b6752, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:07.912435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e4a6f6-2557-44b4-8a16-38304d01c1d9 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:07.938888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1010/2027] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:07 2022] PUT /identity/v3/projects/56741605cfd34e87878dca1ce623da03/users/b7266c6bbe6e426d89a12a0f5d1b6752/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:07.941141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd9f971-80f4-4690-a8d5-dced51aa34ea None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:07.941647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd9f971-80f4-4690-a8d5-dced51aa34ea None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:07.941738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd9f971-80f4-4690-a8d5-dced51aa34ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:07.979364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3dd9f971-80f4-4690-a8d5-dced51aa34ea None None] MFA Rules not processed for user `b7266c6bbe6e426d89a12a0f5d1b6752`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:07.980637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3dd9f971-80f4-4690-a8d5-dced51aa34ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:08.072739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1018/2028] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:07 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:08.088380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e4a6f6-2557-44b4-8a16-38304d01c1d9 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:08.189148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:08.206035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e4a6f6-2557-44b4-8a16-38304d01c1d9 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:08.214409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d606ae-64e6-4a8b-9154-3384a89ba2e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:08.215240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d606ae-64e6-4a8b-9154-3384a89ba2e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:08.215803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d606ae-64e6-4a8b-9154-3384a89ba2e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:08.216218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d606ae-64e6-4a8b-9154-3384a89ba2e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:08.232621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d606ae-64e6-4a8b-9154-3384a89ba2e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:08.233722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d606ae-64e6-4a8b-9154-3384a89ba2e5 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:08.272586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1011/2029] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:08 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:09.246765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd9f971-80f4-4690-a8d5-dced51aa34ea None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:09.256761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd9f971-80f4-4690-a8d5-dced51aa34ea None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:09.264509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1466469-be52-4332-bd7b-102f1efb85b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:09.265879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1466469-be52-4332-bd7b-102f1efb85b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:09.265879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1466469-be52-4332-bd7b-102f1efb85b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:09.265879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1466469-be52-4332-bd7b-102f1efb85b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:09.339117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1466469-be52-4332-bd7b-102f1efb85b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:09.340621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1466469-be52-4332-bd7b-102f1efb85b0 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:09.351482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1019/2030] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:09 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:18.256742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:19.346550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=93 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:22.161451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d606ae-64e6-4a8b-9154-3384a89ba2e5 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:22.171153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d606ae-64e6-4a8b-9154-3384a89ba2e5 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:22.175404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805aa7ca-82bb-4303-8178-d3751e6705e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:22.176261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805aa7ca-82bb-4303-8178-d3751e6705e4 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:22.176543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805aa7ca-82bb-4303-8178-d3751e6705e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:22.176802 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805aa7ca-82bb-4303-8178-d3751e6705e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:22.243312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805aa7ca-82bb-4303-8178-d3751e6705e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:22.244765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805aa7ca-82bb-4303-8178-d3751e6705e4 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:22.256346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1012/2031] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:22 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:31:23.004907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1466469-be52-4332-bd7b-102f1efb85b0 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:23.081167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1466469-be52-4332-bd7b-102f1efb85b0 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:23.086444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:23.087141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:23.087415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:23.087681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] PATH_INFO: `/v3/users/55af8787cc8c4e8bb4b835d6a483d7c1` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:23.088804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=55af8787cc8c4e8bb4b835d6a483d7c1)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:23.090778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:23.142971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55af8787cc8c4e8bb4b835d6a483d7c1'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:23.143654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55af8787cc8c4e8bb4b835d6a483d7c1'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:23.150461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55af8787cc8c4e8bb4b835d6a483d7c1'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:23.151096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55af8787cc8c4e8bb4b835d6a483d7c1'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:23.159788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55af8787cc8c4e8bb4b835d6a483d7c1'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:23.178409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1020/2032] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:23 2022] DELETE /identity/v3/users/55af8787cc8c4e8bb4b835d6a483d7c1 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:23.182935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5f69eb-a9e1-4273-9c2d-ad8831e71be0 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:23.183457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5f69eb-a9e1-4273-9c2d-ad8831e71be0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:23.183457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5f69eb-a9e1-4273-9c2d-ad8831e71be0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:23.204162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-6e5f69eb-a9e1-4273-9c2d-ad8831e71be0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:23.205375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6e5f69eb-a9e1-4273-9c2d-ad8831e71be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:23.266279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1013/2033] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:23.275694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:23.277295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:23.333244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef72caba-888b-462d-a0ff-c6338b871fb0 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:23.337461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45c1adf-6f1c-4294-a7f0-869c6feb4f66 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:23.338066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45c1adf-6f1c-4294-a7f0-869c6feb4f66 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:23.338374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45c1adf-6f1c-4294-a7f0-869c6feb4f66 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:23.338639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45c1adf-6f1c-4294-a7f0-869c6feb4f66 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:23.349113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45c1adf-6f1c-4294-a7f0-869c6feb4f66 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:23.350655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45c1adf-6f1c-4294-a7f0-869c6feb4f66 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:23.361261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1021/2034] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:31:23.642174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5f69eb-a9e1-4273-9c2d-ad8831e71be0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:23.644067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6e5f69eb-a9e1-4273-9c2d-ad8831e71be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:23.657437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5f69eb-a9e1-4273-9c2d-ad8831e71be0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:23.664043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:23.664791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:23.665063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:23.665778 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] PATH_INFO: `/v3/projects/d78f523a266a403bb42f78dcfbaa68ed` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:23.667096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d78f523a266a403bb42f78dcfbaa68ed)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:23.669347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:23.724776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd78f523a266a403bb42f78dcfbaa68ed', 'request_id': 'req-21d9b0f4-81e5-4eef-9119-667031e8b428'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:23.731538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd78f523a266a403bb42f78dcfbaa68ed', 'request_id': 'req-21d9b0f4-81e5-4eef-9119-667031e8b428'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:23.741207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd78f523a266a403bb42f78dcfbaa68ed', 'request_id': 'req-21d9b0f4-81e5-4eef-9119-667031e8b428'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:23.753601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] The token cache is being invalidate because project d78f523a266a403bb42f78dcfbaa68ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:23.754079 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:23.754774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-21d9b0f4-81e5-4eef-9119-667031e8b428 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:23.756361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1014/2035] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:23 2022] DELETE /identity/v3/projects/d78f523a266a403bb42f78dcfbaa68ed => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:23.761373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de66283-cf0d-49a9-9c2a-a43cfc19d61e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:23.761689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de66283-cf0d-49a9-9c2a-a43cfc19d61e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:23.761955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de66283-cf0d-49a9-9c2a-a43cfc19d61e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:23.783036 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-9de66283-cf0d-49a9-9c2a-a43cfc19d61e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:23.785114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9de66283-cf0d-49a9-9c2a-a43cfc19d61e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:23.808828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:23.857256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1022/2036] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:23.865320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5dab70-b769-4e1d-ae47-6e977f83c178 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:23.865594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5dab70-b769-4e1d-ae47-6e977f83c178 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:23.865856 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5dab70-b769-4e1d-ae47-6e977f83c178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:23.885841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-ad5dab70-b769-4e1d-ae47-6e977f83c178 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:23.886983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ad5dab70-b769-4e1d-ae47-6e977f83c178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:23.895251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1015/2037] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:23.899289 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de66283-cf0d-49a9-9c2a-a43cfc19d61e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:23.908838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de66283-cf0d-49a9-9c2a-a43cfc19d61e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:23.913492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adab48a-41fd-4c33-b039-f2d9ef1cdeed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:23.914184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adab48a-41fd-4c33-b039-f2d9ef1cdeed admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:23.914523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adab48a-41fd-4c33-b039-f2d9ef1cdeed admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:23.914794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adab48a-41fd-4c33-b039-f2d9ef1cdeed admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:23.915581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adab48a-41fd-4c33-b039-f2d9ef1cdeed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:23.915943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adab48a-41fd-4c33-b039-f2d9ef1cdeed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:23.917637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adab48a-41fd-4c33-b039-f2d9ef1cdeed admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:23.930588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1023/2038] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:31:23.935156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5dab70-b769-4e1d-ae47-6e977f83c178 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:23.944852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5dab70-b769-4e1d-ae47-6e977f83c178 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:23.949579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4fd620-0a81-4e40-b6bd-78c9c6fbbd4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:23.950371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4fd620-0a81-4e40-b6bd-78c9c6fbbd4f admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:23.950704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4fd620-0a81-4e40-b6bd-78c9c6fbbd4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:23.951037 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4fd620-0a81-4e40-b6bd-78c9c6fbbd4f admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:23.951735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4fd620-0a81-4e40-b6bd-78c9c6fbbd4f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:23.954127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4fd620-0a81-4e40-b6bd-78c9c6fbbd4f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:23.970224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:23.992583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1016/2039] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:23 2022] POST /identity/v3/projects => generated 373 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:23.996840 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adab48a-41fd-4c33-b039-f2d9ef1cdeed admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:24.011219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adab48a-41fd-4c33-b039-f2d9ef1cdeed admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:24.073110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa76f5c9-4074-49dd-97b0-cf7635ce0fee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:24.073740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa76f5c9-4074-49dd-97b0-cf7635ce0fee admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:24.074018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa76f5c9-4074-49dd-97b0-cf7635ce0fee admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:24.074318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa76f5c9-4074-49dd-97b0-cf7635ce0fee admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:24.074955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa76f5c9-4074-49dd-97b0-cf7635ce0fee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:24.076883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa76f5c9-4074-49dd-97b0-cf7635ce0fee admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:24.111912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1024/2040] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:23 2022] POST /identity/v3/users => generated 287 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:24.115878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4fd620-0a81-4e40-b6bd-78c9c6fbbd4f admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:24.126714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4fd620-0a81-4e40-b6bd-78c9c6fbbd4f admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:24.131951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d3650-0588-455b-98d5-34d5c5e294ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:24.132621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d3650-0588-455b-98d5-34d5c5e294ba admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:24.132892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d3650-0588-455b-98d5-34d5c5e294ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:24.133179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d3650-0588-455b-98d5-34d5c5e294ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:24.133830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112d3650-0588-455b-98d5-34d5c5e294ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:24.135722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112d3650-0588-455b-98d5-34d5c5e294ba admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:24.146280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1017/2041] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:24.150414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa76f5c9-4074-49dd-97b0-cf7635ce0fee admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:24.167714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa76f5c9-4074-49dd-97b0-cf7635ce0fee admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:24.174586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3c883a-5b72-4b2d-9315-3d85649455b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:24.175220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3c883a-5b72-4b2d-9315-3d85649455b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:24.175489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3c883a-5b72-4b2d-9315-3d85649455b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:24.175762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3c883a-5b72-4b2d-9315-3d85649455b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:24.176467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3c883a-5b72-4b2d-9315-3d85649455b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:24.178393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3c883a-5b72-4b2d-9315-3d85649455b5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:24.194734 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1025/2042] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:24 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:24.198873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d3650-0588-455b-98d5-34d5c5e294ba admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:24.213474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d3650-0588-455b-98d5-34d5c5e294ba admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:24.224956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eaae29-faab-401f-990b-db845bc6d409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:24.226557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eaae29-faab-401f-990b-db845bc6d409 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:24.226939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eaae29-faab-401f-990b-db845bc6d409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:24.227470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eaae29-faab-401f-990b-db845bc6d409 admin admin] PATH_INFO: `/v3/projects/c1b581d18eee429cb4415287c1887fa7/users/47bc6dff55e847ba93eb7bfdcf4e95b2/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:24.245027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eaae29-faab-401f-990b-db845bc6d409 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1b581d18eee429cb4415287c1887fa7, user_id=47bc6dff55e847ba93eb7bfdcf4e95b2, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:24.247886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eaae29-faab-401f-990b-db845bc6d409 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:24.263345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1018/2043] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:24 2022] PUT /identity/v3/projects/c1b581d18eee429cb4415287c1887fa7/users/47bc6dff55e847ba93eb7bfdcf4e95b2/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:24.274175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1de594a-8508-462f-9f32-76f7986b5eb9 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:24.274621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1de594a-8508-462f-9f32-76f7986b5eb9 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:24.274995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1de594a-8508-462f-9f32-76f7986b5eb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:24.312791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e1de594a-8508-462f-9f32-76f7986b5eb9 None None] MFA Rules not processed for user `47bc6dff55e847ba93eb7bfdcf4e95b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:24.314163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e1de594a-8508-462f-9f32-76f7986b5eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:24.402405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1026/2044] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:24.413972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eaae29-faab-401f-990b-db845bc6d409 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:24.416726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c9eaae29-faab-401f-990b-db845bc6d409 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:24.523227 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eaae29-faab-401f-990b-db845bc6d409 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:24.527604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38df6bef-e368-4682-9d5a-1453f5b1c572 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:24.528250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38df6bef-e368-4682-9d5a-1453f5b1c572 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:24.528569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38df6bef-e368-4682-9d5a-1453f5b1c572 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:24.528840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38df6bef-e368-4682-9d5a-1453f5b1c572 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:24.542676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38df6bef-e368-4682-9d5a-1453f5b1c572 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:24.544115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38df6bef-e368-4682-9d5a-1453f5b1c572 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:24.553224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1019/2045] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:24 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:31:24.625011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1de594a-8508-462f-9f32-76f7986b5eb9 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:24.626468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e1de594a-8508-462f-9f32-76f7986b5eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:24.649266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:24.695794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1de594a-8508-462f-9f32-76f7986b5eb9 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:24.700649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:24.701287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:24.701585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:24.701855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] PATH_INFO: `/v3/users/85e97eae4f084f6cb298dd1a98bdbfa2` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:24.703335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] RBAC: Authorizing `identity:delete_user(user_id=85e97eae4f084f6cb298dd1a98bdbfa2)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:24.705243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:24.753161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85e97eae4f084f6cb298dd1a98bdbfa2'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:24.753807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85e97eae4f084f6cb298dd1a98bdbfa2'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:24.761748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85e97eae4f084f6cb298dd1a98bdbfa2'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:24.762406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85e97eae4f084f6cb298dd1a98bdbfa2'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:24.771293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85e97eae4f084f6cb298dd1a98bdbfa2'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:24.796923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1027/2046] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:24 2022] DELETE /identity/v3/users/85e97eae4f084f6cb298dd1a98bdbfa2 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:24.799005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3dfcc5-c616-4b3a-946f-60a718983e8d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:24.804213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3dfcc5-c616-4b3a-946f-60a718983e8d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:24.804677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3dfcc5-c616-4b3a-946f-60a718983e8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:24.827552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-9b3dfcc5-c616-4b3a-946f-60a718983e8d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:24.830372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9b3dfcc5-c616-4b3a-946f-60a718983e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:24.898473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1020/2047] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:24.908613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:24.910055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:24.966696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426760fd-0606-494a-8485-1fe9aa10480a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:24.970340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6371f04a-0154-4ab3-8757-a32165bb5f71 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:24.971291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6371f04a-0154-4ab3-8757-a32165bb5f71 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:24.971562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6371f04a-0154-4ab3-8757-a32165bb5f71 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:24.971826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6371f04a-0154-4ab3-8757-a32165bb5f71 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:24.981081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6371f04a-0154-4ab3-8757-a32165bb5f71 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:24.982545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6371f04a-0154-4ab3-8757-a32165bb5f71 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:24.992025 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1028/2048] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:25.262393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3dfcc5-c616-4b3a-946f-60a718983e8d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:25.264287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9b3dfcc5-c616-4b3a-946f-60a718983e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:25.278693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3dfcc5-c616-4b3a-946f-60a718983e8d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:25.285222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:25.285898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:25.286208 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:25.286489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] PATH_INFO: `/v3/projects/fd5ad15b266e4ea58300b5dff91e037f` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:25.290888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd5ad15b266e4ea58300b5dff91e037f)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:25.292803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:25.342647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd5ad15b266e4ea58300b5dff91e037f', 'request_id': 'req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:25.348829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:INSERT=2,keystone:SELECT=92,keystone:DELETE=6 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:25.350774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd5ad15b266e4ea58300b5dff91e037f', 'request_id': 'req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:25.361409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd5ad15b266e4ea58300b5dff91e037f', 'request_id': 'req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:25.373809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] The token cache is being invalidate because project fd5ad15b266e4ea58300b5dff91e037f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:25.374150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:25.375017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:25.377250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1021/2049] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:25 2022] DELETE /identity/v3/projects/fd5ad15b266e4ea58300b5dff91e037f => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:25.381483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6371f04a-0154-4ab3-8757-a32165bb5f71 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:25.391392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6371f04a-0154-4ab3-8757-a32165bb5f71 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:25.435915 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:25.455231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:25.455873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:25.456168 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:25.456441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] PATH_INFO: `/v3/users/c19b90915e8c46998afaecfc32d9f9ec` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:25.457547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c19b90915e8c46998afaecfc32d9f9ec)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:25.459517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:25.507718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c19b90915e8c46998afaecfc32d9f9ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:25.508303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c19b90915e8c46998afaecfc32d9f9ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:25.515117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c19b90915e8c46998afaecfc32d9f9ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:25.515759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c19b90915e8c46998afaecfc32d9f9ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:25.525130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c19b90915e8c46998afaecfc32d9f9ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:25.545262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1029/2050] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:25 2022] DELETE /identity/v3/users/c19b90915e8c46998afaecfc32d9f9ec => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:25.945687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:25.947364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:25.971517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00b02bb-efc5-4db5-bd0b-0a8889057a1b admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:26.038685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:26.039470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:26.039755 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:26.040030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] PATH_INFO: `/v3/projects/062dc9d61b5b4bab93d847b7b4011cca` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:26.041213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=062dc9d61b5b4bab93d847b7b4011cca)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:26.043245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:26.092015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '062dc9d61b5b4bab93d847b7b4011cca', 'request_id': 'req-65598aed-9bfe-416e-b36c-65f5705d45fe'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.098086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '062dc9d61b5b4bab93d847b7b4011cca', 'request_id': 'req-65598aed-9bfe-416e-b36c-65f5705d45fe'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.108929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '062dc9d61b5b4bab93d847b7b4011cca', 'request_id': 'req-65598aed-9bfe-416e-b36c-65f5705d45fe'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.120678 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] The token cache is being invalidate because project 062dc9d61b5b4bab93d847b7b4011cca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:26.120960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.121592 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.123142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1022/2051] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:25 2022] DELETE /identity/v3/projects/062dc9d61b5b4bab93d847b7b4011cca => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:26.128042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:26.129546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:26.143166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63823db8-874d-4ebb-84b0-9f7059703d85 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:26.211409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:26.212331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:26.212744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:26.213144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] PATH_INFO: `/v3/users/b7266c6bbe6e426d89a12a0f5d1b6752` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:26.214843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7266c6bbe6e426d89a12a0f5d1b6752)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:26.217832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:26.278837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7266c6bbe6e426d89a12a0f5d1b6752'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.279484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7266c6bbe6e426d89a12a0f5d1b6752'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.286315 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7266c6bbe6e426d89a12a0f5d1b6752'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.286934 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7266c6bbe6e426d89a12a0f5d1b6752'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.290584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:26.297906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-04ff52c2-1d13-45ad-96a3-4db6b7e510f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7266c6bbe6e426d89a12a0f5d1b6752'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.316008 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1030/2052] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:26 2022] DELETE /identity/v3/users/b7266c6bbe6e426d89a12a0f5d1b6752 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:26.597111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:26.598748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:26.615252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65598aed-9bfe-416e-b36c-65f5705d45fe admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:26.680800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:26.681479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:26.681733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:26.682003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] PATH_INFO: `/v3/projects/56741605cfd34e87878dca1ce623da03` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:26.683141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=56741605cfd34e87878dca1ce623da03)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:26.685045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:26.701553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:26.734324 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56741605cfd34e87878dca1ce623da03', 'request_id': 'req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.739970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56741605cfd34e87878dca1ce623da03', 'request_id': 'req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.749938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56741605cfd34e87878dca1ce623da03', 'request_id': 'req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.761934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] The token cache is being invalidate because project 56741605cfd34e87878dca1ce623da03 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:26.762254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.762974 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12ad6104-7e3d-4bf8-bfb1-ea41a74bfb4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:26.764585 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1023/2053] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:26 2022] DELETE /identity/v3/projects/56741605cfd34e87878dca1ce623da03 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:26.770737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e974831-80d7-4b28-95cb-c803c39e715c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:26.771173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e974831-80d7-4b28-95cb-c803c39e715c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:26.771276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e974831-80d7-4b28-95cb-c803c39e715c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:26.791714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2e974831-80d7-4b28-95cb-c803c39e715c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:26.792923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2e974831-80d7-4b28-95cb-c803c39e715c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:26.852947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1031/2054] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:26.861493 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a9f33b-d31f-4013-a645-f46c9f33d9d0 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:26.862022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a9f33b-d31f-4013-a645-f46c9f33d9d0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:26.862454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a9f33b-d31f-4013-a645-f46c9f33d9d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:26.883081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c0a9f33b-d31f-4013-a645-f46c9f33d9d0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:26.884401 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c0a9f33b-d31f-4013-a645-f46c9f33d9d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:26.893412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1024/2055] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:26.897733 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e974831-80d7-4b28-95cb-c803c39e715c None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:26.909046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e974831-80d7-4b28-95cb-c803c39e715c None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:26.913939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db16832-c557-4bba-952e-f7e7765bcaf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:26.914576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db16832-c557-4bba-952e-f7e7765bcaf5 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:26.914852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db16832-c557-4bba-952e-f7e7765bcaf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:26.915212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db16832-c557-4bba-952e-f7e7765bcaf5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:26.915942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db16832-c557-4bba-952e-f7e7765bcaf5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:26.916246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db16832-c557-4bba-952e-f7e7765bcaf5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:26.917856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db16832-c557-4bba-952e-f7e7765bcaf5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:26.930436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1032/2056] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:26.934840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a9f33b-d31f-4013-a645-f46c9f33d9d0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:26.944375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a9f33b-d31f-4013-a645-f46c9f33d9d0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:26.949221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be9ed5a-50dd-4acb-bcba-757cdbe06f70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:26.949834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be9ed5a-50dd-4acb-bcba-757cdbe06f70 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:26.950122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be9ed5a-50dd-4acb-bcba-757cdbe06f70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:26.950400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be9ed5a-50dd-4acb-bcba-757cdbe06f70 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:26.951037 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be9ed5a-50dd-4acb-bcba-757cdbe06f70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:26.953280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be9ed5a-50dd-4acb-bcba-757cdbe06f70 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:26.982302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1025/2057] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:26 2022] POST /identity/v3/projects => generated 379 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:31:26.986597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db16832-c557-4bba-952e-f7e7765bcaf5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:26.996757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db16832-c557-4bba-952e-f7e7765bcaf5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.053908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c8b4ef-e8cd-40ac-b4ad-f72df78d8c77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.054547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c8b4ef-e8cd-40ac-b4ad-f72df78d8c77 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.054827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c8b4ef-e8cd-40ac-b4ad-f72df78d8c77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.055096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c8b4ef-e8cd-40ac-b4ad-f72df78d8c77 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.055723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c8b4ef-e8cd-40ac-b4ad-f72df78d8c77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.057658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c8b4ef-e8cd-40ac-b4ad-f72df78d8c77 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:27.087913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1033/2058] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:26 2022] POST /identity/v3/users => generated 290 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:27.089074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be9ed5a-50dd-4acb-bcba-757cdbe06f70 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:27.100025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be9ed5a-50dd-4acb-bcba-757cdbe06f70 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.105745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e849a34a-c3e2-4691-bbfb-f6deb6a650d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.106312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e849a34a-c3e2-4691-bbfb-f6deb6a650d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.106587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e849a34a-c3e2-4691-bbfb-f6deb6a650d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.106854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e849a34a-c3e2-4691-bbfb-f6deb6a650d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.107520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e849a34a-c3e2-4691-bbfb-f6deb6a650d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.109223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e849a34a-c3e2-4691-bbfb-f6deb6a650d2 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:27.125178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1026/2059] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:27 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:27.130918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c8b4ef-e8cd-40ac-b4ad-f72df78d8c77 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:27.145983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c8b4ef-e8cd-40ac-b4ad-f72df78d8c77 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.153768 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249a14a9-0394-4157-b0ce-32a49616f2ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.154646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a14a9-0394-4157-b0ce-32a49616f2ba admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.154993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a14a9-0394-4157-b0ce-32a49616f2ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.155295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a14a9-0394-4157-b0ce-32a49616f2ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.156109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249a14a9-0394-4157-b0ce-32a49616f2ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.157723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249a14a9-0394-4157-b0ce-32a49616f2ba admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:27.168512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1034/2060] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:27 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:27.174295 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e849a34a-c3e2-4691-bbfb-f6deb6a650d2 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:27.188765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e849a34a-c3e2-4691-bbfb-f6deb6a650d2 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.193754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a362509d-ace6-4cc6-a263-3abde08a11eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.194421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a362509d-ace6-4cc6-a263-3abde08a11eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.194701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a362509d-ace6-4cc6-a263-3abde08a11eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.194970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a362509d-ace6-4cc6-a263-3abde08a11eb admin admin] PATH_INFO: `/v3/projects/c7067ab2fbd84bbdbb59af9cddc60987/users/e1be2c85a7244387bfda6a3e89bacf2d/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.210387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a362509d-ace6-4cc6-a263-3abde08a11eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7067ab2fbd84bbdbb59af9cddc60987, user_id=e1be2c85a7244387bfda6a3e89bacf2d, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.213024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a362509d-ace6-4cc6-a263-3abde08a11eb admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:27.228132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1027/2061] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:27 2022] PUT /identity/v3/projects/c7067ab2fbd84bbdbb59af9cddc60987/users/e1be2c85a7244387bfda6a3e89bacf2d/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:27.241015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92bfef0-182f-4459-9c39-cf27d6744b83 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.241447 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92bfef0-182f-4459-9c39-cf27d6744b83 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.241863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92bfef0-182f-4459-9c39-cf27d6744b83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.277785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-b92bfef0-182f-4459-9c39-cf27d6744b83 None None] MFA Rules not processed for user `e1be2c85a7244387bfda6a3e89bacf2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:27.279054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b92bfef0-182f-4459-9c39-cf27d6744b83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:27.312572 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:27.348794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1035/2062] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:27 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:27.353512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a362509d-ace6-4cc6-a263-3abde08a11eb admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:27.416107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a362509d-ace6-4cc6-a263-3abde08a11eb admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.421674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bc9417-985a-4f8d-baa0-6d3d3b04732b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.422312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bc9417-985a-4f8d-baa0-6d3d3b04732b admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.422579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bc9417-985a-4f8d-baa0-6d3d3b04732b admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.422851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bc9417-985a-4f8d-baa0-6d3d3b04732b admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.423491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bc9417-985a-4f8d-baa0-6d3d3b04732b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.425557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bc9417-985a-4f8d-baa0-6d3d3b04732b admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:27.455729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1028/2063] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:27 2022] POST /identity/v3/projects => generated 381 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:27.459981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92bfef0-182f-4459-9c39-cf27d6744b83 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:27.471390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92bfef0-182f-4459-9c39-cf27d6744b83 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.529719 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d225ac92-a0c3-4098-8761-b0af631760c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.530351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d225ac92-a0c3-4098-8761-b0af631760c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.530649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d225ac92-a0c3-4098-8761-b0af631760c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.531062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d225ac92-a0c3-4098-8761-b0af631760c5 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.531681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d225ac92-a0c3-4098-8761-b0af631760c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.533626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d225ac92-a0c3-4098-8761-b0af631760c5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:27.557804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1036/2064] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:27 2022] POST /identity/v3/users => generated 297 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:27.562128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bc9417-985a-4f8d-baa0-6d3d3b04732b admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:27.572525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bc9417-985a-4f8d-baa0-6d3d3b04732b admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.577158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f67adb-d7e4-4833-bd15-eecb3a0791a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.577756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f67adb-d7e4-4833-bd15-eecb3a0791a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.578032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f67adb-d7e4-4833-bd15-eecb3a0791a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.578439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f67adb-d7e4-4833-bd15-eecb3a0791a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.579087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f67adb-d7e4-4833-bd15-eecb3a0791a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.580701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f67adb-d7e4-4833-bd15-eecb3a0791a1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:27.591128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1029/2065] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:27.596019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d225ac92-a0c3-4098-8761-b0af631760c5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:27.609284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d225ac92-a0c3-4098-8761-b0af631760c5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.614626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125cb3c6-8453-4940-84d4-2333e4750393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.615396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125cb3c6-8453-4940-84d4-2333e4750393 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.615731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125cb3c6-8453-4940-84d4-2333e4750393 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.616005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125cb3c6-8453-4940-84d4-2333e4750393 admin admin] PATH_INFO: `/v3/projects/4dc09430353c4ecba96d6fb93e3e89a1/users/d8b2e9d935414b0fbd3b0f95af99f85c/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.635294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125cb3c6-8453-4940-84d4-2333e4750393 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4dc09430353c4ecba96d6fb93e3e89a1, user_id=d8b2e9d935414b0fbd3b0f95af99f85c, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.638838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125cb3c6-8453-4940-84d4-2333e4750393 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:27.656509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1037/2066] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:27 2022] PUT /identity/v3/projects/4dc09430353c4ecba96d6fb93e3e89a1/users/d8b2e9d935414b0fbd3b0f95af99f85c/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:27.660905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f67adb-d7e4-4833-bd15-eecb3a0791a1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:27.671627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f67adb-d7e4-4833-bd15-eecb3a0791a1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.683296 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:27.731803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c298a0e1-14f6-4d39-81da-ff4a96c3a2f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.732414 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c298a0e1-14f6-4d39-81da-ff4a96c3a2f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.732712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c298a0e1-14f6-4d39-81da-ff4a96c3a2f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.732984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c298a0e1-14f6-4d39-81da-ff4a96c3a2f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.733782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c298a0e1-14f6-4d39-81da-ff4a96c3a2f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.735538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c298a0e1-14f6-4d39-81da-ff4a96c3a2f2 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:27.746682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1030/2067] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:27 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:27.751212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125cb3c6-8453-4940-84d4-2333e4750393 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:27.762807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125cb3c6-8453-4940-84d4-2333e4750393 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.768563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd195805-1ada-4a99-b9a3-2640fd11151e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.769266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd195805-1ada-4a99-b9a3-2640fd11151e admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.769541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd195805-1ada-4a99-b9a3-2640fd11151e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.769813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd195805-1ada-4a99-b9a3-2640fd11151e admin admin] PATH_INFO: `/v3/projects/4dc09430353c4ecba96d6fb93e3e89a1/users/d8b2e9d935414b0fbd3b0f95af99f85c/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.771814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd195805-1ada-4a99-b9a3-2640fd11151e admin admin] RBAC: Authorizing `identity:create_grant(project_id=4dc09430353c4ecba96d6fb93e3e89a1, user_id=d8b2e9d935414b0fbd3b0f95af99f85c, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.774537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd195805-1ada-4a99-b9a3-2640fd11151e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:27.787337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1038/2068] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:27 2022] PUT /identity/v3/projects/4dc09430353c4ecba96d6fb93e3e89a1/users/d8b2e9d935414b0fbd3b0f95af99f85c/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:27.799053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c766c95-c2a0-40bc-bcbf-e2bcb75dd7df None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.799351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c766c95-c2a0-40bc-bcbf-e2bcb75dd7df None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.799627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c766c95-c2a0-40bc-bcbf-e2bcb75dd7df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.830904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8c766c95-c2a0-40bc-bcbf-e2bcb75dd7df None None] MFA Rules not processed for user `d8b2e9d935414b0fbd3b0f95af99f85c`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:27.832125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8c766c95-c2a0-40bc-bcbf-e2bcb75dd7df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:27.905001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1031/2069] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:27 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:27.916343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd195805-1ada-4a99-b9a3-2640fd11151e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:27.918393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-dd195805-1ada-4a99-b9a3-2640fd11151e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:27.982589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd195805-1ada-4a99-b9a3-2640fd11151e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:27.986385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0f7654-10aa-4a93-9fc4-66c60721a2c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:27.987001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0f7654-10aa-4a93-9fc4-66c60721a2c3 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:27.987304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0f7654-10aa-4a93-9fc4-66c60721a2c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:27.987577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0f7654-10aa-4a93-9fc4-66c60721a2c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:27.997595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0f7654-10aa-4a93-9fc4-66c60721a2c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:27.999319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0f7654-10aa-4a93-9fc4-66c60721a2c3 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:28.009460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1039/2070] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:27 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:28.424799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c766c95-c2a0-40bc-bcbf-e2bcb75dd7df None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:28.492439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c766c95-c2a0-40bc-bcbf-e2bcb75dd7df None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:28.497554 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:28.498221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:28.498500 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:28.498770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] PATH_INFO: `/v3/users/e1be2c85a7244387bfda6a3e89bacf2d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:28.501058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1be2c85a7244387bfda6a3e89bacf2d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:28.503065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:28.548194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:28.557981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1be2c85a7244387bfda6a3e89bacf2d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:28.568942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1be2c85a7244387bfda6a3e89bacf2d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:28.569680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1be2c85a7244387bfda6a3e89bacf2d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:28.570451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1be2c85a7244387bfda6a3e89bacf2d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:28.578935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1be2c85a7244387bfda6a3e89bacf2d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:28.597418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1032/2071] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:28 2022] DELETE /identity/v3/users/e1be2c85a7244387bfda6a3e89bacf2d => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:28.602202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f7bd38-bb63-46bd-8bbb-6f6b1b4a1e66 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:28.602489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f7bd38-bb63-46bd-8bbb-6f6b1b4a1e66 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:28.602770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f7bd38-bb63-46bd-8bbb-6f6b1b4a1e66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:28.622894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-b7f7bd38-bb63-46bd-8bbb-6f6b1b4a1e66 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:28.624096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b7f7bd38-bb63-46bd-8bbb-6f6b1b4a1e66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:28.649310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:28.687091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1040/2072] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:28.698221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:28.700400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:28.761397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7397c2d0-5b2f-4712-a0c2-c4e1e261842f admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:28.765618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3b2a43-b5fa-4488-b115-0f372a924e2e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:28.766275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3b2a43-b5fa-4488-b115-0f372a924e2e service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:28.766534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3b2a43-b5fa-4488-b115-0f372a924e2e service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:28.766799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3b2a43-b5fa-4488-b115-0f372a924e2e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:28.775838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3b2a43-b5fa-4488-b115-0f372a924e2e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:28.777276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3b2a43-b5fa-4488-b115-0f372a924e2e service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:28.786659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1033/2073] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:29.168066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f7bd38-bb63-46bd-8bbb-6f6b1b4a1e66 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:29.169504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b7f7bd38-bb63-46bd-8bbb-6f6b1b4a1e66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:29.194412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f7bd38-bb63-46bd-8bbb-6f6b1b4a1e66 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:29.199713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:29.200497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:29.200987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:29.201355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] PATH_INFO: `/v3/projects/c7067ab2fbd84bbdbb59af9cddc60987` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:29.203187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c7067ab2fbd84bbdbb59af9cddc60987)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:29.205921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:29.273601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c7067ab2fbd84bbdbb59af9cddc60987', 'request_id': 'req-a5193973-0e7f-4462-ab83-82defd1908f6'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:29.286429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c7067ab2fbd84bbdbb59af9cddc60987', 'request_id': 'req-a5193973-0e7f-4462-ab83-82defd1908f6'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:29.303289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b0fe51-6d4e-4fbf-970b-b008b73232ce None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:29.303573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b0fe51-6d4e-4fbf-970b-b008b73232ce None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:29.303847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b0fe51-6d4e-4fbf-970b-b008b73232ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:29.306202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c7067ab2fbd84bbdbb59af9cddc60987', 'request_id': 'req-a5193973-0e7f-4462-ab83-82defd1908f6'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:29.325584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] The token cache is being invalidate because project c7067ab2fbd84bbdbb59af9cddc60987 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:29.325880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:29.326607 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:29.328456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1041/2074] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:29 2022] DELETE /identity/v3/projects/c7067ab2fbd84bbdbb59af9cddc60987 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:29.330543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b7b0fe51-6d4e-4fbf-970b-b008b73232ce None None] MFA Rules not processed for user `47bc6dff55e847ba93eb7bfdcf4e95b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:29.332731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:29.333516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b7b0fe51-6d4e-4fbf-970b-b008b73232ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:29.335516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:29.413609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1034/2075] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:29.428439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5193973-0e7f-4462-ab83-82defd1908f6 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:29.433201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:29.433806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:29.434821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:29.434821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] PATH_INFO: `/v3/users/d8b2e9d935414b0fbd3b0f95af99f85c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:29.435772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8b2e9d935414b0fbd3b0f95af99f85c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:29.437807 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:29.497643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8b2e9d935414b0fbd3b0f95af99f85c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:29.498346 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8b2e9d935414b0fbd3b0f95af99f85c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:29.505526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:29.508057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8b2e9d935414b0fbd3b0f95af99f85c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:29.508799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8b2e9d935414b0fbd3b0f95af99f85c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:29.520526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d9b2c66e-f7ae-4a7a-94da-7d7a596e77b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8b2e9d935414b0fbd3b0f95af99f85c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:29.548085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1042/2076] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:29 2022] DELETE /identity/v3/users/d8b2e9d935414b0fbd3b0f95af99f85c => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:29.857536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b0fe51-6d4e-4fbf-970b-b008b73232ce None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:29.877814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b0fe51-6d4e-4fbf-970b-b008b73232ce None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:29.941676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:29.942799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:29.942799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:29.942964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] PATH_INFO: `/v3/projects/4dc09430353c4ecba96d6fb93e3e89a1` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:29.944149 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4dc09430353c4ecba96d6fb93e3e89a1)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:29.946242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:29.955543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:30.015936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4dc09430353c4ecba96d6fb93e3e89a1', 'request_id': 'req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:30.021631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4dc09430353c4ecba96d6fb93e3e89a1', 'request_id': 'req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:30.030274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4dc09430353c4ecba96d6fb93e3e89a1', 'request_id': 'req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:30.042327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] The token cache is being invalidate because project 4dc09430353c4ecba96d6fb93e3e89a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:30.042614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:30.043245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-70e1a5e7-ab14-4319-9a5c-9a84b29fba15 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:30.045195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1035/2077] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:29 2022] DELETE /identity/v3/projects/4dc09430353c4ecba96d6fb93e3e89a1 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:30.050358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9ef66b-339b-4cad-a45a-8d51e2c21a92 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.050584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9ef66b-339b-4cad-a45a-8d51e2c21a92 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.050852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9ef66b-339b-4cad-a45a-8d51e2c21a92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.070783 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-6a9ef66b-339b-4cad-a45a-8d51e2c21a92 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:30.072008 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6a9ef66b-339b-4cad-a45a-8d51e2c21a92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:30.146273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1043/2078] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:30.149213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e037c32-5fe4-4c85-9281-44d74fecb5aa None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.149501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e037c32-5fe4-4c85-9281-44d74fecb5aa None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.149788 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e037c32-5fe4-4c85-9281-44d74fecb5aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.171084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0e037c32-5fe4-4c85-9281-44d74fecb5aa None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:30.172947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0e037c32-5fe4-4c85-9281-44d74fecb5aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:30.181860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1036/2079] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:30.186144 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9ef66b-339b-4cad-a45a-8d51e2c21a92 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:30.204973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9ef66b-339b-4cad-a45a-8d51e2c21a92 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:30.210920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aeebf4-0c9a-4c99-8bec-2906d4a5d78c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:30.211538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aeebf4-0c9a-4c99-8bec-2906d4a5d78c admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.212174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aeebf4-0c9a-4c99-8bec-2906d4a5d78c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.212174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aeebf4-0c9a-4c99-8bec-2906d4a5d78c admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.212810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3aeebf4-0c9a-4c99-8bec-2906d4a5d78c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:30.213175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3aeebf4-0c9a-4c99-8bec-2906d4a5d78c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:30.215312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3aeebf4-0c9a-4c99-8bec-2906d4a5d78c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:30.236301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1044/2080] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:30.240476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e037c32-5fe4-4c85-9281-44d74fecb5aa None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:30.252452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e037c32-5fe4-4c85-9281-44d74fecb5aa None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:30.257949 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9c9256-1d52-43b0-8885-430f42d9846d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:30.258627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c9256-1d52-43b0-8885-430f42d9846d admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.258902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c9256-1d52-43b0-8885-430f42d9846d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.259211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9c9256-1d52-43b0-8885-430f42d9846d admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.259816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9c9256-1d52-43b0-8885-430f42d9846d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:30.261994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9c9256-1d52-43b0-8885-430f42d9846d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:30.305219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1037/2081] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:30 2022] POST /identity/v3/projects => generated 383 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:30.306608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aeebf4-0c9a-4c99-8bec-2906d4a5d78c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:30.317605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aeebf4-0c9a-4c99-8bec-2906d4a5d78c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:30.391812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307382da-cb16-4130-a74f-1634e5d7bc4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:30.392792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307382da-cb16-4130-a74f-1634e5d7bc4f admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.393133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307382da-cb16-4130-a74f-1634e5d7bc4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.393464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307382da-cb16-4130-a74f-1634e5d7bc4f admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.395722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307382da-cb16-4130-a74f-1634e5d7bc4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:30.396511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307382da-cb16-4130-a74f-1634e5d7bc4f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:30.425090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1045/2082] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:30 2022] POST /identity/v3/users => generated 292 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:30.429366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9c9256-1d52-43b0-8885-430f42d9846d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:30.454610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9c9256-1d52-43b0-8885-430f42d9846d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:30.459319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6dbdbf-6f10-4362-9c5d-8e3007a6f1b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:30.459925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6dbdbf-6f10-4362-9c5d-8e3007a6f1b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.460197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6dbdbf-6f10-4362-9c5d-8e3007a6f1b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.460460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6dbdbf-6f10-4362-9c5d-8e3007a6f1b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.461107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6dbdbf-6f10-4362-9c5d-8e3007a6f1b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:30.462816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6dbdbf-6f10-4362-9c5d-8e3007a6f1b4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:30.477917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1038/2083] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:30 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:30.479305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307382da-cb16-4130-a74f-1634e5d7bc4f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:30.490213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307382da-cb16-4130-a74f-1634e5d7bc4f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:30.496159 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd05515-06b0-44c0-bb6d-e6e634d3f016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:30.496665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd05515-06b0-44c0-bb6d-e6e634d3f016 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.496941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd05515-06b0-44c0-bb6d-e6e634d3f016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.497207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd05515-06b0-44c0-bb6d-e6e634d3f016 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.497849 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd05515-06b0-44c0-bb6d-e6e634d3f016 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:30.499513 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd05515-06b0-44c0-bb6d-e6e634d3f016 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:30.513550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1046/2084] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:30 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:30.517581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6dbdbf-6f10-4362-9c5d-8e3007a6f1b4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:30.532991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6dbdbf-6f10-4362-9c5d-8e3007a6f1b4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:30.538455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d50b21-b58f-4865-8f92-06a76652247e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:30.539123 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d50b21-b58f-4865-8f92-06a76652247e admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.539453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d50b21-b58f-4865-8f92-06a76652247e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.539645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d50b21-b58f-4865-8f92-06a76652247e admin admin] PATH_INFO: `/v3/projects/7891230751e14cbb97ce52b41516ac7a/users/f14f2d93afc34e3ea8488793bb2e2b1d/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.555790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d50b21-b58f-4865-8f92-06a76652247e admin admin] RBAC: Authorizing `identity:create_grant(project_id=7891230751e14cbb97ce52b41516ac7a, user_id=f14f2d93afc34e3ea8488793bb2e2b1d, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:30.558345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d50b21-b58f-4865-8f92-06a76652247e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:30.574729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1039/2085] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:30 2022] PUT /identity/v3/projects/7891230751e14cbb97ce52b41516ac7a/users/f14f2d93afc34e3ea8488793bb2e2b1d/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:30.586657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4611b7e5-1119-4d37-95c3-3fb55c511bb1 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.586657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4611b7e5-1119-4d37-95c3-3fb55c511bb1 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.586948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4611b7e5-1119-4d37-95c3-3fb55c511bb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.621593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-4611b7e5-1119-4d37-95c3-3fb55c511bb1 None None] MFA Rules not processed for user `f14f2d93afc34e3ea8488793bb2e2b1d`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:30.622883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4611b7e5-1119-4d37-95c3-3fb55c511bb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:30.658443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:30.697422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1047/2086] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:30.702068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d50b21-b58f-4865-8f92-06a76652247e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:30.771328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d50b21-b58f-4865-8f92-06a76652247e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:30.774768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bda336-163d-4102-99aa-dda50f732919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:30.775375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bda336-163d-4102-99aa-dda50f732919 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.775650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bda336-163d-4102-99aa-dda50f732919 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.775918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bda336-163d-4102-99aa-dda50f732919 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.776575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bda336-163d-4102-99aa-dda50f732919 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:30.778509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bda336-163d-4102-99aa-dda50f732919 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:30.813447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1040/2087] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:30 2022] POST /identity/v3/projects => generated 385 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:30.817415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4611b7e5-1119-4d37-95c3-3fb55c511bb1 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:30.830235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4611b7e5-1119-4d37-95c3-3fb55c511bb1 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:30.890663 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d9983c-42fd-4332-93eb-e0a85a98469e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:30.894286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d9983c-42fd-4332-93eb-e0a85a98469e admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.894286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d9983c-42fd-4332-93eb-e0a85a98469e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.894286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d9983c-42fd-4332-93eb-e0a85a98469e admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.894286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d9983c-42fd-4332-93eb-e0a85a98469e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:30.894683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d9983c-42fd-4332-93eb-e0a85a98469e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:30.919048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1048/2088] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:30 2022] POST /identity/v3/users => generated 299 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:31:30.923650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bda336-163d-4102-99aa-dda50f732919 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:30.934810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bda336-163d-4102-99aa-dda50f732919 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:30.943902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d989e9-166e-4095-8418-c01a1b9785a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:30.944601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d989e9-166e-4095-8418-c01a1b9785a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.944871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d989e9-166e-4095-8418-c01a1b9785a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.945137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d989e9-166e-4095-8418-c01a1b9785a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:30.945782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d989e9-166e-4095-8418-c01a1b9785a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:30.947483 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d989e9-166e-4095-8418-c01a1b9785a4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:30.958834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1041/2089] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:30.964720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d9983c-42fd-4332-93eb-e0a85a98469e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:30.977108 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d9983c-42fd-4332-93eb-e0a85a98469e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:30.984687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b2e374-9dc1-43d0-b8ce-c4d469b1279b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:30.985348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b2e374-9dc1-43d0-b8ce-c4d469b1279b admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:30.985622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b2e374-9dc1-43d0-b8ce-c4d469b1279b admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:30.985910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b2e374-9dc1-43d0-b8ce-c4d469b1279b admin admin] PATH_INFO: `/v3/projects/a06937f5ee144a97a9441cdf8a466bbc/users/cebe9a655f544da5971c3f9d91e8283d/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:31.001475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b2e374-9dc1-43d0-b8ce-c4d469b1279b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a06937f5ee144a97a9441cdf8a466bbc, user_id=cebe9a655f544da5971c3f9d91e8283d, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:31.004212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b2e374-9dc1-43d0-b8ce-c4d469b1279b admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:31.038335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1049/2090] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:30 2022] PUT /identity/v3/projects/a06937f5ee144a97a9441cdf8a466bbc/users/cebe9a655f544da5971c3f9d91e8283d/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:31.042622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d989e9-166e-4095-8418-c01a1b9785a4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:31.049738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:31.054175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d989e9-166e-4095-8418-c01a1b9785a4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:31.126437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94027f69-a6fb-4afa-b4d3-917927314706 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:31.127063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94027f69-a6fb-4afa-b4d3-917927314706 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:31.127341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94027f69-a6fb-4afa-b4d3-917927314706 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:31.127613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94027f69-a6fb-4afa-b4d3-917927314706 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:31.128264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94027f69-a6fb-4afa-b4d3-917927314706 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:31.129905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94027f69-a6fb-4afa-b4d3-917927314706 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:31.140596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1042/2091] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:31 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:31.145269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b2e374-9dc1-43d0-b8ce-c4d469b1279b admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:31.156328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b2e374-9dc1-43d0-b8ce-c4d469b1279b admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:31.162061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d2b1fa-21c8-4acb-9126-f9177e9481eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:31.162735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d2b1fa-21c8-4acb-9126-f9177e9481eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:31.163017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d2b1fa-21c8-4acb-9126-f9177e9481eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:31.163286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d2b1fa-21c8-4acb-9126-f9177e9481eb admin admin] PATH_INFO: `/v3/projects/a06937f5ee144a97a9441cdf8a466bbc/users/cebe9a655f544da5971c3f9d91e8283d/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:31.165450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d2b1fa-21c8-4acb-9126-f9177e9481eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=a06937f5ee144a97a9441cdf8a466bbc, user_id=cebe9a655f544da5971c3f9d91e8283d, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:31.168336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d2b1fa-21c8-4acb-9126-f9177e9481eb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:31.180125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1050/2092] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:31 2022] PUT /identity/v3/projects/a06937f5ee144a97a9441cdf8a466bbc/users/cebe9a655f544da5971c3f9d91e8283d/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:31.191070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aeb41b-a070-4c71-b517-7eee8c830a48 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:31.191354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aeb41b-a070-4c71-b517-7eee8c830a48 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:31.191583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aeb41b-a070-4c71-b517-7eee8c830a48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:31.220840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-40aeb41b-a070-4c71-b517-7eee8c830a48 None None] MFA Rules not processed for user `cebe9a655f544da5971c3f9d91e8283d`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:31.222056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-40aeb41b-a070-4c71-b517-7eee8c830a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:31.298073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1043/2093] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:31 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:31.309321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d2b1fa-21c8-4acb-9126-f9177e9481eb admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:31.311432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-97d2b1fa-21c8-4acb-9126-f9177e9481eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:31.373812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d2b1fa-21c8-4acb-9126-f9177e9481eb admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:31.377364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83328817-ed43-4b4a-b34c-dba529247e76 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:31.377966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83328817-ed43-4b4a-b34c-dba529247e76 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:31.378271 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83328817-ed43-4b4a-b34c-dba529247e76 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:31.378545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83328817-ed43-4b4a-b34c-dba529247e76 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:31.410338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:31.440764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83328817-ed43-4b4a-b34c-dba529247e76 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:31.442483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83328817-ed43-4b4a-b34c-dba529247e76 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:31.451924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1051/2094] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:33.421223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aeb41b-a070-4c71-b517-7eee8c830a48 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:33.422945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-40aeb41b-a070-4c71-b517-7eee8c830a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:33.493879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aeb41b-a070-4c71-b517-7eee8c830a48 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:33.498568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:33.499213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:33.499488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:33.500006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] PATH_INFO: `/v3/users/2045e938699844569c177d0f37229fe3` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:33.500997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2045e938699844569c177d0f37229fe3)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:33.502957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:33.542124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:33.570614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2045e938699844569c177d0f37229fe3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:33.580255 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2045e938699844569c177d0f37229fe3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:33.581027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2045e938699844569c177d0f37229fe3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:33.581686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2045e938699844569c177d0f37229fe3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:33.588734 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2045e938699844569c177d0f37229fe3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:33.606360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1044/2095] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:33 2022] DELETE /identity/v3/users/2045e938699844569c177d0f37229fe3 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:33.611217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24c19e0-36bb-4848-b1ba-3ed827802841 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:33.611676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24c19e0-36bb-4848-b1ba-3ed827802841 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:33.611765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24c19e0-36bb-4848-b1ba-3ed827802841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:33.632183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d24c19e0-36bb-4848-b1ba-3ed827802841 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:33.633406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d24c19e0-36bb-4848-b1ba-3ed827802841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:33.692232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1052/2096] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:33.702453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:33.704181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:33.762957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea99685-7683-4c93-8f57-35d8415e78d5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:33.767213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d47f32-e764-4e65-a0fb-827c8083376e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:33.767818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d47f32-e764-4e65-a0fb-827c8083376e service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:33.768099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d47f32-e764-4e65-a0fb-827c8083376e service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:33.768367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d47f32-e764-4e65-a0fb-827c8083376e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:33.777643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d47f32-e764-4e65-a0fb-827c8083376e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:33.779092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d47f32-e764-4e65-a0fb-827c8083376e service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:33.788595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1045/2097] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:31:34.073074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24c19e0-36bb-4848-b1ba-3ed827802841 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:34.074851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d24c19e0-36bb-4848-b1ba-3ed827802841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:34.088453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24c19e0-36bb-4848-b1ba-3ed827802841 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:34.093729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:34.094519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:34.094787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:34.095117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] PATH_INFO: `/v3/projects/9f2a675882b549238f66b56ab1aa21af` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:34.097066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f2a675882b549238f66b56ab1aa21af)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:34.100269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:34.150971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f2a675882b549238f66b56ab1aa21af', 'request_id': 'req-17181ba5-5bf9-4f62-916e-a4557336366d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:34.157076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f2a675882b549238f66b56ab1aa21af', 'request_id': 'req-17181ba5-5bf9-4f62-916e-a4557336366d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:34.169714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f2a675882b549238f66b56ab1aa21af', 'request_id': 'req-17181ba5-5bf9-4f62-916e-a4557336366d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:34.183218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] The token cache is being invalidate because project 9f2a675882b549238f66b56ab1aa21af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:34.183801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:34.184319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-17181ba5-5bf9-4f62-916e-a4557336366d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:34.186135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1053/2098] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:34 2022] DELETE /identity/v3/projects/9f2a675882b549238f66b56ab1aa21af => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:34.191328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e08ede3-12ac-4561-9841-7d98eb5be97b None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:34.191604 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e08ede3-12ac-4561-9841-7d98eb5be97b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:34.191900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e08ede3-12ac-4561-9841-7d98eb5be97b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:34.214677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-1e08ede3-12ac-4561-9841-7d98eb5be97b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:34.216056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1e08ede3-12ac-4561-9841-7d98eb5be97b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:34.281038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1046/2099] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:34.290472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5d0752-2179-4638-bc02-dbb0e5c73684 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:34.290785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5d0752-2179-4638-bc02-dbb0e5c73684 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:34.291050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5d0752-2179-4638-bc02-dbb0e5c73684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:34.335286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-5c5d0752-2179-4638-bc02-dbb0e5c73684 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:34.337386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-5c5d0752-2179-4638-bc02-dbb0e5c73684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:34.350009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1054/2100] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:34.362445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e08ede3-12ac-4561-9841-7d98eb5be97b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:34.376523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e08ede3-12ac-4561-9841-7d98eb5be97b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:34.378700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26565589-ae36-4955-9980-c5aeacc9da6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:34.379316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26565589-ae36-4955-9980-c5aeacc9da6e admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:34.379587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26565589-ae36-4955-9980-c5aeacc9da6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:34.379853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26565589-ae36-4955-9980-c5aeacc9da6e admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:34.380499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26565589-ae36-4955-9980-c5aeacc9da6e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:34.380796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26565589-ae36-4955-9980-c5aeacc9da6e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:34.382488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26565589-ae36-4955-9980-c5aeacc9da6e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:34.439271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1047/2101] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:34.445127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5d0752-2179-4638-bc02-dbb0e5c73684 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:34.465519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5d0752-2179-4638-bc02-dbb0e5c73684 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:34.472453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dee2245-a3eb-400a-9b3a-c0fff597cb3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:34.473080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dee2245-a3eb-400a-9b3a-c0fff597cb3e admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:34.473530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dee2245-a3eb-400a-9b3a-c0fff597cb3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:34.473707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dee2245-a3eb-400a-9b3a-c0fff597cb3e admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:34.474379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dee2245-a3eb-400a-9b3a-c0fff597cb3e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:34.477235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dee2245-a3eb-400a-9b3a-c0fff597cb3e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:34.514294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1055/2102] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:34 2022] POST /identity/v3/projects => generated 377 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:34.517256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26565589-ae36-4955-9980-c5aeacc9da6e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:34.527134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26565589-ae36-4955-9980-c5aeacc9da6e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:34.567547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:34.612923 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25df1342-0b45-4621-a1a3-b098e55a30a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:34.614684 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25df1342-0b45-4621-a1a3-b098e55a30a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:34.614969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25df1342-0b45-4621-a1a3-b098e55a30a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:34.615309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25df1342-0b45-4621-a1a3-b098e55a30a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:34.616033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25df1342-0b45-4621-a1a3-b098e55a30a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:34.617963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25df1342-0b45-4621-a1a3-b098e55a30a8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:34.646167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1048/2103] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:34 2022] POST /identity/v3/users => generated 289 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:34.650130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dee2245-a3eb-400a-9b3a-c0fff597cb3e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:34.661556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dee2245-a3eb-400a-9b3a-c0fff597cb3e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:34.666563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5410858c-de9a-47e9-93e7-c2a11b51ca00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:34.667162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5410858c-de9a-47e9-93e7-c2a11b51ca00 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:34.667434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5410858c-de9a-47e9-93e7-c2a11b51ca00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:34.667701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5410858c-de9a-47e9-93e7-c2a11b51ca00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:34.668340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5410858c-de9a-47e9-93e7-c2a11b51ca00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:34.669936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5410858c-de9a-47e9-93e7-c2a11b51ca00 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:34.683661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1056/2104] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:34.684820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25df1342-0b45-4621-a1a3-b098e55a30a8 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:34.696680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25df1342-0b45-4621-a1a3-b098e55a30a8 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:34.709286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e745ba-4734-497e-8a54-79509a25f98b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:34.710055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e745ba-4734-497e-8a54-79509a25f98b admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:34.710445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e745ba-4734-497e-8a54-79509a25f98b admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:34.710805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e745ba-4734-497e-8a54-79509a25f98b admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:34.711718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e745ba-4734-497e-8a54-79509a25f98b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:34.713578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e745ba-4734-497e-8a54-79509a25f98b admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:34.728831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1049/2105] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:34 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:31:34.730189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5410858c-de9a-47e9-93e7-c2a11b51ca00 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:34.741070 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5410858c-de9a-47e9-93e7-c2a11b51ca00 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:34.746060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed9d77b-f36b-43ac-b1a6-cebbbc783240 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:34.747003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed9d77b-f36b-43ac-b1a6-cebbbc783240 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:34.747223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed9d77b-f36b-43ac-b1a6-cebbbc783240 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:34.747408 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed9d77b-f36b-43ac-b1a6-cebbbc783240 admin admin] PATH_INFO: `/v3/projects/93213d315669449086d549efa0cecb5b/users/7074f76be84d49da8df2400162f49ebf/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:34.768164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed9d77b-f36b-43ac-b1a6-cebbbc783240 admin admin] RBAC: Authorizing `identity:create_grant(project_id=93213d315669449086d549efa0cecb5b, user_id=7074f76be84d49da8df2400162f49ebf, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:34.772914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed9d77b-f36b-43ac-b1a6-cebbbc783240 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:34.775425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:34.795952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1057/2106] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:34 2022] PUT /identity/v3/projects/93213d315669449086d549efa0cecb5b/users/7074f76be84d49da8df2400162f49ebf/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:34.801436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f14962b-7543-4a8c-8878-10578f795faf None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:34.801756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f14962b-7543-4a8c-8878-10578f795faf None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:34.802116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f14962b-7543-4a8c-8878-10578f795faf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:34.831497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-2f14962b-7543-4a8c-8878-10578f795faf None None] MFA Rules not processed for user `7074f76be84d49da8df2400162f49ebf`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:34.832704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2f14962b-7543-4a8c-8878-10578f795faf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:34.905140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1050/2107] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:34 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:34.910554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed9d77b-f36b-43ac-b1a6-cebbbc783240 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:34.994214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed9d77b-f36b-43ac-b1a6-cebbbc783240 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:34.999842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fa36a1-a17a-4748-807b-7f753c8f6f9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.000459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fa36a1-a17a-4748-807b-7f753c8f6f9b admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.000738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fa36a1-a17a-4748-807b-7f753c8f6f9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.001008 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fa36a1-a17a-4748-807b-7f753c8f6f9b admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.002284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fa36a1-a17a-4748-807b-7f753c8f6f9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.004273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fa36a1-a17a-4748-807b-7f753c8f6f9b admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.033680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1058/2108] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:34 2022] POST /identity/v3/projects => generated 377 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:35.037849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f14962b-7543-4a8c-8878-10578f795faf None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.049729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f14962b-7543-4a8c-8878-10578f795faf None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:35.117757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d58cf1-5119-480d-a925-36148f101c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.118450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d58cf1-5119-480d-a925-36148f101c83 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.118686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d58cf1-5119-480d-a925-36148f101c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.118925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d58cf1-5119-480d-a925-36148f101c83 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.119565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d58cf1-5119-480d-a925-36148f101c83 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.122237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d58cf1-5119-480d-a925-36148f101c83 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.151036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1051/2109] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:35 2022] POST /identity/v3/users => generated 289 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:35.154980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fa36a1-a17a-4748-807b-7f753c8f6f9b admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.166993 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fa36a1-a17a-4748-807b-7f753c8f6f9b admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:35.172379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab4e497-4742-45d5-a352-c1bf85d3e045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.173011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab4e497-4742-45d5-a352-c1bf85d3e045 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.173386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab4e497-4742-45d5-a352-c1bf85d3e045 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.173654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab4e497-4742-45d5-a352-c1bf85d3e045 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.174414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab4e497-4742-45d5-a352-c1bf85d3e045 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.176047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab4e497-4742-45d5-a352-c1bf85d3e045 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.187737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1059/2110] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:35 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:35.192595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d58cf1-5119-480d-a925-36148f101c83 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.206085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d58cf1-5119-480d-a925-36148f101c83 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:35.208143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c45985-464e-445e-9d2b-4a87159bf55e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.208754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c45985-464e-445e-9d2b-4a87159bf55e admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.209051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c45985-464e-445e-9d2b-4a87159bf55e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.209317 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c45985-464e-445e-9d2b-4a87159bf55e admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.209961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c45985-464e-445e-9d2b-4a87159bf55e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.211636 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c45985-464e-445e-9d2b-4a87159bf55e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.223339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1052/2111] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:35.229874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab4e497-4742-45d5-a352-c1bf85d3e045 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.248190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab4e497-4742-45d5-a352-c1bf85d3e045 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:35.253342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3001bfae-e19e-4b43-b2c6-0d97af322d6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.253998 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3001bfae-e19e-4b43-b2c6-0d97af322d6e admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.254308 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3001bfae-e19e-4b43-b2c6-0d97af322d6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.254592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3001bfae-e19e-4b43-b2c6-0d97af322d6e admin admin] PATH_INFO: `/v3/projects/6040624371194ff9907f09ae3e684938/users/9de6299fd85d446aabe26c6d8eb829e2/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.272335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3001bfae-e19e-4b43-b2c6-0d97af322d6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6040624371194ff9907f09ae3e684938, user_id=9de6299fd85d446aabe26c6d8eb829e2, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.275047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3001bfae-e19e-4b43-b2c6-0d97af322d6e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.291921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1060/2112] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:35 2022] PUT /identity/v3/projects/6040624371194ff9907f09ae3e684938/users/9de6299fd85d446aabe26c6d8eb829e2/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:35.302579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e241ce5b-cd39-4708-9e46-1aa25577c325 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.302864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e241ce5b-cd39-4708-9e46-1aa25577c325 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.303247 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e241ce5b-cd39-4708-9e46-1aa25577c325 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.338542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e241ce5b-cd39-4708-9e46-1aa25577c325 None None] MFA Rules not processed for user `9de6299fd85d446aabe26c6d8eb829e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:35.339808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e241ce5b-cd39-4708-9e46-1aa25577c325 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:35.353910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:35.439021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1053/2113] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:35 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:35.443717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3001bfae-e19e-4b43-b2c6-0d97af322d6e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.523778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3001bfae-e19e-4b43-b2c6-0d97af322d6e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:35.529763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb21b506-f41d-4ccd-a59b-162343b93126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.531182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb21b506-f41d-4ccd-a59b-162343b93126 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.534068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb21b506-f41d-4ccd-a59b-162343b93126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.534376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb21b506-f41d-4ccd-a59b-162343b93126 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.536155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb21b506-f41d-4ccd-a59b-162343b93126 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.538378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb21b506-f41d-4ccd-a59b-162343b93126 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.577300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1061/2114] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:35 2022] POST /identity/v3/projects => generated 377 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:35.581592 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e241ce5b-cd39-4708-9e46-1aa25577c325 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.597649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e241ce5b-cd39-4708-9e46-1aa25577c325 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:35.669878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83582265-5555-4779-a979-b6b4a4380910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.671195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83582265-5555-4779-a979-b6b4a4380910 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.671558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83582265-5555-4779-a979-b6b4a4380910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.671960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83582265-5555-4779-a979-b6b4a4380910 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.672795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83582265-5555-4779-a979-b6b4a4380910 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.676103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83582265-5555-4779-a979-b6b4a4380910 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.706895 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1054/2115] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:35 2022] POST /identity/v3/users => generated 295 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:35.711150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb21b506-f41d-4ccd-a59b-162343b93126 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.723195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb21b506-f41d-4ccd-a59b-162343b93126 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:35.728385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d46fe0-008d-45dd-bcbf-4a4f11460213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.729033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d46fe0-008d-45dd-bcbf-4a4f11460213 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.729305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d46fe0-008d-45dd-bcbf-4a4f11460213 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.729569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d46fe0-008d-45dd-bcbf-4a4f11460213 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.730387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d46fe0-008d-45dd-bcbf-4a4f11460213 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.731901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d46fe0-008d-45dd-bcbf-4a4f11460213 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.744931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1062/2116] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:35.750726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83582265-5555-4779-a979-b6b4a4380910 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.761455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83582265-5555-4779-a979-b6b4a4380910 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:35.766821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b38e8a4-ce60-4cd3-a0c2-f336e23690e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.767856 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b38e8a4-ce60-4cd3-a0c2-f336e23690e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.768139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b38e8a4-ce60-4cd3-a0c2-f336e23690e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.768577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b38e8a4-ce60-4cd3-a0c2-f336e23690e4 admin admin] PATH_INFO: `/v3/projects/90980e4d215d43a5924a03aec1268de7/users/4c70e54e4b3b407295a2d0c9ecb459bc/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.790476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b38e8a4-ce60-4cd3-a0c2-f336e23690e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=90980e4d215d43a5924a03aec1268de7, user_id=4c70e54e4b3b407295a2d0c9ecb459bc, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.794182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b38e8a4-ce60-4cd3-a0c2-f336e23690e4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.817060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1055/2117] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:35 2022] PUT /identity/v3/projects/90980e4d215d43a5924a03aec1268de7/users/4c70e54e4b3b407295a2d0c9ecb459bc/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:35.821258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d46fe0-008d-45dd-bcbf-4a4f11460213 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.833241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d46fe0-008d-45dd-bcbf-4a4f11460213 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:35.852038 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:35.905939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a24490-c892-4166-8ced-c1e9b2a2d85d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.906599 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a24490-c892-4166-8ced-c1e9b2a2d85d admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.906877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a24490-c892-4166-8ced-c1e9b2a2d85d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.907404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a24490-c892-4166-8ced-c1e9b2a2d85d admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.908496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b38e8a4-ce60-4cd3-a0c2-f336e23690e4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.908618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a24490-c892-4166-8ced-c1e9b2a2d85d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.910424 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a24490-c892-4166-8ced-c1e9b2a2d85d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.910696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2b38e8a4-ce60-4cd3-a0c2-f336e23690e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:35.923054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1063/2118] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:35 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:31:35.927254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a24490-c892-4166-8ced-c1e9b2a2d85d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:35.940637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a24490-c892-4166-8ced-c1e9b2a2d85d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:35.946634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f724cdc-b898-42d9-98e7-bed83bc7ac5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:35.947292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f724cdc-b898-42d9-98e7-bed83bc7ac5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.947586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f724cdc-b898-42d9-98e7-bed83bc7ac5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.947855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f724cdc-b898-42d9-98e7-bed83bc7ac5d admin admin] PATH_INFO: `/v3/projects/90980e4d215d43a5924a03aec1268de7/users/4c70e54e4b3b407295a2d0c9ecb459bc/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.950185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f724cdc-b898-42d9-98e7-bed83bc7ac5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=90980e4d215d43a5924a03aec1268de7, user_id=4c70e54e4b3b407295a2d0c9ecb459bc, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:35.952878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f724cdc-b898-42d9-98e7-bed83bc7ac5d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:35.966639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1064/2119] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:35 2022] PUT /identity/v3/projects/90980e4d215d43a5924a03aec1268de7/users/4c70e54e4b3b407295a2d0c9ecb459bc/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:35.978353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f700750-cb41-4416-a77f-28d9c9779954 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:35.978718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f700750-cb41-4416-a77f-28d9c9779954 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:35.979054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f700750-cb41-4416-a77f-28d9c9779954 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:35.980681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b38e8a4-ce60-4cd3-a0c2-f336e23690e4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:36.012202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-6f700750-cb41-4416-a77f-28d9c9779954 None None] MFA Rules not processed for user `4c70e54e4b3b407295a2d0c9ecb459bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:36.013521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6f700750-cb41-4416-a77f-28d9c9779954 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:36.019175 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:36.043383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d983434c-6fe0-4610-b058-4f61224cbfed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:36.044001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d983434c-6fe0-4610-b058-4f61224cbfed service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:36.044268 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d983434c-6fe0-4610-b058-4f61224cbfed service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:36.044532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d983434c-6fe0-4610-b058-4f61224cbfed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:36.046315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d983434c-6fe0-4610-b058-4f61224cbfed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:36.097360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1065/2120] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:35 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:36.111188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f700750-cb41-4416-a77f-28d9c9779954 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:36.113446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6f700750-cb41-4416-a77f-28d9c9779954 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:36.118028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d983434c-6fe0-4610-b058-4f61224cbfed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:36.119705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d983434c-6fe0-4610-b058-4f61224cbfed service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:36.126775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f700750-cb41-4416-a77f-28d9c9779954 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:36.129538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1056/2121] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:35 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:31:36.131183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86f7741-0599-42b9-95c3-d2c9553de0a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:36.131900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86f7741-0599-42b9-95c3-d2c9553de0a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:36.132245 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86f7741-0599-42b9-95c3-d2c9553de0a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:36.132624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86f7741-0599-42b9-95c3-d2c9553de0a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:36.194539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86f7741-0599-42b9-95c3-d2c9553de0a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:36.196028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86f7741-0599-42b9-95c3-d2c9553de0a5 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:36.205940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1066/2122] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:36 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:39.007008 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d983434c-6fe0-4610-b058-4f61224cbfed service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:39.016781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d983434c-6fe0-4610-b058-4f61224cbfed service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:39.028719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db865cb-abe8-4b38-ad6e-efb6b4ba645a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:39.029446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db865cb-abe8-4b38-ad6e-efb6b4ba645a service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:39.029793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db865cb-abe8-4b38-ad6e-efb6b4ba645a service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:39.030163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db865cb-abe8-4b38-ad6e-efb6b4ba645a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:39.105397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db865cb-abe8-4b38-ad6e-efb6b4ba645a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:39.106956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db865cb-abe8-4b38-ad6e-efb6b4ba645a service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:39.117436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1057/2123] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:39 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:39.631996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f18be57-e04a-4e44-badc-c38ed5a3410a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:39.632856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f18be57-e04a-4e44-badc-c38ed5a3410a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:39.633213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f18be57-e04a-4e44-badc-c38ed5a3410a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:39.668413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7f18be57-e04a-4e44-badc-c38ed5a3410a None None] MFA Rules not processed for user `5ae72d51e9004670a50362d1206fab21`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:39.670058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7f18be57-e04a-4e44-badc-c38ed5a3410a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:39.695710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:39.741436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1067/2124] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:39.881697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db865cb-abe8-4b38-ad6e-efb6b4ba645a service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:39.890472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db865cb-abe8-4b38-ad6e-efb6b4ba645a service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:39.895764 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90daf68-97dd-44f8-9c71-a755211ef457 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:39.896488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90daf68-97dd-44f8-9c71-a755211ef457 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:39.896853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90daf68-97dd-44f8-9c71-a755211ef457 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:39.897212 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90daf68-97dd-44f8-9c71-a755211ef457 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:39.918171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90daf68-97dd-44f8-9c71-a755211ef457 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:39.918171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90daf68-97dd-44f8-9c71-a755211ef457 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:39.922981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1058/2125] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:39 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:41.271266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f18be57-e04a-4e44-badc-c38ed5a3410a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:41.273014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7f18be57-e04a-4e44-badc-c38ed5a3410a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:41.384483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f18be57-e04a-4e44-badc-c38ed5a3410a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:41.395397 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:41.396072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:41.396352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:41.396628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] PATH_INFO: `/v3/users/f14f2d93afc34e3ea8488793bb2e2b1d` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:41.398180 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f14f2d93afc34e3ea8488793bb2e2b1d)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:41.400146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:41.454956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f14f2d93afc34e3ea8488793bb2e2b1d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:41.455624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f14f2d93afc34e3ea8488793bb2e2b1d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:41.463335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f14f2d93afc34e3ea8488793bb2e2b1d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:41.464300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f14f2d93afc34e3ea8488793bb2e2b1d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:41.474557 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f14f2d93afc34e3ea8488793bb2e2b1d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:41.495290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1068/2126] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:41 2022] DELETE /identity/v3/users/f14f2d93afc34e3ea8488793bb2e2b1d => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:41.500032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca44f7c-0a31-437c-b74d-601f86247f57 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:41.500523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca44f7c-0a31-437c-b74d-601f86247f57 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:41.500630 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca44f7c-0a31-437c-b74d-601f86247f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:41.521066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-cca44f7c-0a31-437c-b74d-601f86247f57 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:41.522424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-cca44f7c-0a31-437c-b74d-601f86247f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:41.587857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1059/2127] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:41.599143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:41.600570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:41.663806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:41.668113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32763919-4f6b-4783-90a9-b48f92df81a1 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:41.673431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a0f695-a370-42e8-8a8d-3654a5f13299 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:41.674443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a0f695-a370-42e8-8a8d-3654a5f13299 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:41.674964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a0f695-a370-42e8-8a8d-3654a5f13299 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:41.675457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a0f695-a370-42e8-8a8d-3654a5f13299 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:41.685175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a0f695-a370-42e8-8a8d-3654a5f13299 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:41.686670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a0f695-a370-42e8-8a8d-3654a5f13299 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:41.696462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1069/2128] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:41.979209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca44f7c-0a31-437c-b74d-601f86247f57 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:41.980702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-cca44f7c-0a31-437c-b74d-601f86247f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:42.002490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca44f7c-0a31-437c-b74d-601f86247f57 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:42.007943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:42.015663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:42.016019 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:42.016300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] PATH_INFO: `/v3/projects/7891230751e14cbb97ce52b41516ac7a` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:42.017687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7891230751e14cbb97ce52b41516ac7a)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:42.019763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:42.022375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:42.072371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7891230751e14cbb97ce52b41516ac7a', 'request_id': 'req-9357c052-16e7-41df-a327-e83d4900a74b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.080934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7891230751e14cbb97ce52b41516ac7a', 'request_id': 'req-9357c052-16e7-41df-a327-e83d4900a74b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.092011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7891230751e14cbb97ce52b41516ac7a', 'request_id': 'req-9357c052-16e7-41df-a327-e83d4900a74b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.105395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] The token cache is being invalidate because project 7891230751e14cbb97ce52b41516ac7a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:42.105693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.106366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.110762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1060/2129] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:41 2022] DELETE /identity/v3/projects/7891230751e14cbb97ce52b41516ac7a => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:42.112188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a0f695-a370-42e8-8a8d-3654a5f13299 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:42.122191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a0f695-a370-42e8-8a8d-3654a5f13299 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:42.181430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:42.182403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:42.182803 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:42.183085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] PATH_INFO: `/v3/users/cebe9a655f544da5971c3f9d91e8283d` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:42.184248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cebe9a655f544da5971c3f9d91e8283d)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:42.186718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:42.237244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cebe9a655f544da5971c3f9d91e8283d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.238364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cebe9a655f544da5971c3f9d91e8283d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.248224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cebe9a655f544da5971c3f9d91e8283d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.264139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cebe9a655f544da5971c3f9d91e8283d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.270525 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-40f5a293-8b3a-4ea0-a787-62fdf71c8b18 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cebe9a655f544da5971c3f9d91e8283d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.290385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1070/2130] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:42 2022] DELETE /identity/v3/users/cebe9a655f544da5971c3f9d91e8283d => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:42.687574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:42.689439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:42.706911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9357c052-16e7-41df-a327-e83d4900a74b admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:42.777324 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:42.777990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:42.778306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:42.778582 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] PATH_INFO: `/v3/projects/a06937f5ee144a97a9441cdf8a466bbc` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:42.781677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a06937f5ee144a97a9441cdf8a466bbc)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:42.783941 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:42.883714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a06937f5ee144a97a9441cdf8a466bbc', 'request_id': 'req-c36bd38c-19b5-489e-b529-b7dde14bdc9f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.892803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a06937f5ee144a97a9441cdf8a466bbc', 'request_id': 'req-c36bd38c-19b5-489e-b529-b7dde14bdc9f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.906504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a06937f5ee144a97a9441cdf8a466bbc', 'request_id': 'req-c36bd38c-19b5-489e-b529-b7dde14bdc9f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.933319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] The token cache is being invalidate because project a06937f5ee144a97a9441cdf8a466bbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:42.933608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.934390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c36bd38c-19b5-489e-b529-b7dde14bdc9f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:42.936124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1061/2131] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:42 2022] DELETE /identity/v3/projects/a06937f5ee144a97a9441cdf8a466bbc => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:42.943909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be79b2a-0256-431b-b9d2-5d1f149987f3 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:42.944409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be79b2a-0256-431b-b9d2-5d1f149987f3 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:42.944688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be79b2a-0256-431b-b9d2-5d1f149987f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:42.972037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-6be79b2a-0256-431b-b9d2-5d1f149987f3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:42.973290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6be79b2a-0256-431b-b9d2-5d1f149987f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:43.038874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1071/2132] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:43.047638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900dedb3-eb7f-4720-8844-0ed65f1732e9 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:43.048362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900dedb3-eb7f-4720-8844-0ed65f1732e9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:43.048693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900dedb3-eb7f-4720-8844-0ed65f1732e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:43.071053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-900dedb3-eb7f-4720-8844-0ed65f1732e9 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:43.072288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-900dedb3-eb7f-4720-8844-0ed65f1732e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:43.081400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1062/2133] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:43.086354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be79b2a-0256-431b-b9d2-5d1f149987f3 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:43.124205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be79b2a-0256-431b-b9d2-5d1f149987f3 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:43.133966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d5ad73-65ba-460a-8c69-1b62a7dafeec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:43.135011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5ad73-65ba-460a-8c69-1b62a7dafeec admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:43.135581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5ad73-65ba-460a-8c69-1b62a7dafeec admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:43.135936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5ad73-65ba-460a-8c69-1b62a7dafeec admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:43.136984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5ad73-65ba-460a-8c69-1b62a7dafeec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:43.137463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5ad73-65ba-460a-8c69-1b62a7dafeec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:43.140128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5ad73-65ba-460a-8c69-1b62a7dafeec admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:43.152928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:43.164696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1072/2134] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:43.165040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900dedb3-eb7f-4720-8844-0ed65f1732e9 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:43.176677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900dedb3-eb7f-4720-8844-0ed65f1732e9 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:43.188509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff13dbc9-34fa-4b85-a9a7-96760bb0ab5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:43.189153 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff13dbc9-34fa-4b85-a9a7-96760bb0ab5f admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:43.189421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff13dbc9-34fa-4b85-a9a7-96760bb0ab5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:43.189773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff13dbc9-34fa-4b85-a9a7-96760bb0ab5f admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:43.190379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff13dbc9-34fa-4b85-a9a7-96760bb0ab5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:43.214608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff13dbc9-34fa-4b85-a9a7-96760bb0ab5f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:43.243583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:43.272735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1063/2135] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:43 2022] POST /identity/v3/projects => generated 399 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:43.277312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d5ad73-65ba-460a-8c69-1b62a7dafeec admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:43.292680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d5ad73-65ba-460a-8c69-1b62a7dafeec admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:43.413419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81d2f3f-2215-475e-8bd2-5c4766964584 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:43.414186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81d2f3f-2215-475e-8bd2-5c4766964584 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:43.414561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81d2f3f-2215-475e-8bd2-5c4766964584 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:43.414965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81d2f3f-2215-475e-8bd2-5c4766964584 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:43.415682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81d2f3f-2215-475e-8bd2-5c4766964584 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:43.419171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81d2f3f-2215-475e-8bd2-5c4766964584 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:43.462987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1073/2136] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:43 2022] POST /identity/v3/users => generated 300 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:31:43.467272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff13dbc9-34fa-4b85-a9a7-96760bb0ab5f admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:43.479908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff13dbc9-34fa-4b85-a9a7-96760bb0ab5f admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:43.490776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e778249a-1dae-444b-8f6a-bbe9ed5d0f07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:43.491718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e778249a-1dae-444b-8f6a-bbe9ed5d0f07 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:43.492028 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e778249a-1dae-444b-8f6a-bbe9ed5d0f07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:43.492338 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e778249a-1dae-444b-8f6a-bbe9ed5d0f07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:43.493026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e778249a-1dae-444b-8f6a-bbe9ed5d0f07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:43.495239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e778249a-1dae-444b-8f6a-bbe9ed5d0f07 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:43.508911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1064/2137] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:43 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:43.513072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81d2f3f-2215-475e-8bd2-5c4766964584 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:43.528500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81d2f3f-2215-475e-8bd2-5c4766964584 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:43.533303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c769df75-bcd4-4c67-acc2-457f2b3906c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:43.534208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c769df75-bcd4-4c67-acc2-457f2b3906c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:43.534664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c769df75-bcd4-4c67-acc2-457f2b3906c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:43.535176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c769df75-bcd4-4c67-acc2-457f2b3906c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:43.536122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c769df75-bcd4-4c67-acc2-457f2b3906c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:43.538737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c769df75-bcd4-4c67-acc2-457f2b3906c2 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:43.553614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1074/2138] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:43 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:43.557832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e778249a-1dae-444b-8f6a-bbe9ed5d0f07 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:43.584459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e778249a-1dae-444b-8f6a-bbe9ed5d0f07 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:43.589352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873a388a-f34f-441a-8512-c882856bdb57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:43.590410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873a388a-f34f-441a-8512-c882856bdb57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:43.590683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873a388a-f34f-441a-8512-c882856bdb57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:43.590952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873a388a-f34f-441a-8512-c882856bdb57 admin admin] PATH_INFO: `/v3/projects/c4ea71ef4ef9490eb90e3405499df93d/users/b05ed8547e514d858ca26cc7708285a6/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:43.609012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873a388a-f34f-441a-8512-c882856bdb57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4ea71ef4ef9490eb90e3405499df93d, user_id=b05ed8547e514d858ca26cc7708285a6, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:43.611864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873a388a-f34f-441a-8512-c882856bdb57 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:43.631091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1065/2139] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:43 2022] PUT /identity/v3/projects/c4ea71ef4ef9490eb90e3405499df93d/users/b05ed8547e514d858ca26cc7708285a6/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:43.658969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bbf7a8-e42e-4a90-9127-a36c8b38afff None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:43.659478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bbf7a8-e42e-4a90-9127-a36c8b38afff None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:43.659894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bbf7a8-e42e-4a90-9127-a36c8b38afff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:43.691420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-87bbf7a8-e42e-4a90-9127-a36c8b38afff None None] MFA Rules not processed for user `b05ed8547e514d858ca26cc7708285a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:43.692664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-87bbf7a8-e42e-4a90-9127-a36c8b38afff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:43.768109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1075/2140] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:43 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:43.778891 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873a388a-f34f-441a-8512-c882856bdb57 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:43.781033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-873a388a-f34f-441a-8512-c882856bdb57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:43.849213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873a388a-f34f-441a-8512-c882856bdb57 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:43.853067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023e903a-cb88-4e12-bff9-966a6f522674 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:43.853785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023e903a-cb88-4e12-bff9-966a6f522674 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:43.854150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023e903a-cb88-4e12-bff9-966a6f522674 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:43.854495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023e903a-cb88-4e12-bff9-966a6f522674 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:43.866045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023e903a-cb88-4e12-bff9-966a6f522674 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:43.867763 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023e903a-cb88-4e12-bff9-966a6f522674 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:43.880457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1066/2141] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:43 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:44.061391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bbf7a8-e42e-4a90-9127-a36c8b38afff None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:44.062896 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-87bbf7a8-e42e-4a90-9127-a36c8b38afff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:44.155485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bbf7a8-e42e-4a90-9127-a36c8b38afff None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:44.161530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:44.162429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:44.162851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:44.163279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] PATH_INFO: `/v3/users/7074f76be84d49da8df2400162f49ebf` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:44.164557 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7074f76be84d49da8df2400162f49ebf)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:44.167133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:44.173388 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:44.235918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7074f76be84d49da8df2400162f49ebf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:44.236667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7074f76be84d49da8df2400162f49ebf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:44.244614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7074f76be84d49da8df2400162f49ebf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:44.245405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7074f76be84d49da8df2400162f49ebf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:44.259334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7074f76be84d49da8df2400162f49ebf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:44.310915 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1076/2142] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:44 2022] DELETE /identity/v3/users/7074f76be84d49da8df2400162f49ebf => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:44.312879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6164d84-6e00-4f63-af73-952996931b88 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:44.313187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6164d84-6e00-4f63-af73-952996931b88 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:44.313461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6164d84-6e00-4f63-af73-952996931b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:44.362216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c6164d84-6e00-4f63-af73-952996931b88 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:44.363489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c6164d84-6e00-4f63-af73-952996931b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:44.476232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1067/2143] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:44.491734 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:44.494879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:44.583354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a442f9bc-edc6-4924-ae3d-5ad010044d74 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:44.587512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbe6887-8e37-4a48-89dc-9877e2c5b555 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:44.588136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbe6887-8e37-4a48-89dc-9877e2c5b555 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:44.588411 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbe6887-8e37-4a48-89dc-9877e2c5b555 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:44.588680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbe6887-8e37-4a48-89dc-9877e2c5b555 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:44.604694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbe6887-8e37-4a48-89dc-9877e2c5b555 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:44.606247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbe6887-8e37-4a48-89dc-9877e2c5b555 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:44.616240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1077/2144] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:45.198061 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6164d84-6e00-4f63-af73-952996931b88 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:45.201530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c6164d84-6e00-4f63-af73-952996931b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:45.226030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6164d84-6e00-4f63-af73-952996931b88 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:45.242250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:45.243013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:45.243362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:45.243647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] PATH_INFO: `/v3/projects/93213d315669449086d549efa0cecb5b` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:45.245429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=93213d315669449086d549efa0cecb5b)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:45.251052 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:45.358950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93213d315669449086d549efa0cecb5b', 'request_id': 'req-13681c4f-9ee0-4693-8e79-be29ffceb5b2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:45.366216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:INSERT=2,keystone:SELECT=92,keystone:DELETE=6 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:45.368432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93213d315669449086d549efa0cecb5b', 'request_id': 'req-13681c4f-9ee0-4693-8e79-be29ffceb5b2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:45.384054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93213d315669449086d549efa0cecb5b', 'request_id': 'req-13681c4f-9ee0-4693-8e79-be29ffceb5b2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:45.403443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] The token cache is being invalidate because project 93213d315669449086d549efa0cecb5b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:45.403957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:45.404860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:45.406935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1068/2145] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:45 2022] DELETE /identity/v3/projects/93213d315669449086d549efa0cecb5b => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:45.411799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbe6887-8e37-4a48-89dc-9877e2c5b555 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:45.423003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbe6887-8e37-4a48-89dc-9877e2c5b555 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:45.539163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:45.540574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:45.540893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:45.541139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] PATH_INFO: `/v3/users/9de6299fd85d446aabe26c6d8eb829e2` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:45.543387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] RBAC: Authorizing `identity:delete_user(user_id=9de6299fd85d446aabe26c6d8eb829e2)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:45.547367 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:45.643752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:45.710345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9de6299fd85d446aabe26c6d8eb829e2'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:45.712127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9de6299fd85d446aabe26c6d8eb829e2'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:45.737347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9de6299fd85d446aabe26c6d8eb829e2'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:45.739331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9de6299fd85d446aabe26c6d8eb829e2'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:45.767853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9de6299fd85d446aabe26c6d8eb829e2'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:45.811827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1078/2146] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:45 2022] DELETE /identity/v3/users/9de6299fd85d446aabe26c6d8eb829e2 => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:46.185638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:46.187418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:46.207696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13681c4f-9ee0-4693-8e79-be29ffceb5b2 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:46.279200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:46.279200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:46.279200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:46.279200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] PATH_INFO: `/v3/projects/6040624371194ff9907f09ae3e684938` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:46.280349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6040624371194ff9907f09ae3e684938)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:46.280814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:46.329541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6040624371194ff9907f09ae3e684938', 'request_id': 'req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:46.335886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6040624371194ff9907f09ae3e684938', 'request_id': 'req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:46.344362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6040624371194ff9907f09ae3e684938', 'request_id': 'req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:46.357858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] The token cache is being invalidate because project 6040624371194ff9907f09ae3e684938 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:46.358169 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:46.358824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e3d7a85f-395d-4aa3-b796-e8bb5a5689f3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:46.360379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1069/2147] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:46 2022] DELETE /identity/v3/projects/6040624371194ff9907f09ae3e684938 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:46.365158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:46.366636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:46.379737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9c4417-7432-41a5-a34d-e1382aa14cfd admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:46.440970 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:46.442380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:46.442770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:46.443327 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] PATH_INFO: `/v3/users/4c70e54e4b3b407295a2d0c9ecb459bc` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:46.444462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c70e54e4b3b407295a2d0c9ecb459bc)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:46.446551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:46.509992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c70e54e4b3b407295a2d0c9ecb459bc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:46.512488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c70e54e4b3b407295a2d0c9ecb459bc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:46.522515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c70e54e4b3b407295a2d0c9ecb459bc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:46.523918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c70e54e4b3b407295a2d0c9ecb459bc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:46.534965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c70e54e4b3b407295a2d0c9ecb459bc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:46.546897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830abaa4-3f77-4d11-b4e9-59476f317594 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:46.547466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830abaa4-3f77-4d11-b4e9-59476f317594 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:46.548279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830abaa4-3f77-4d11-b4e9-59476f317594 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:46.550073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1070/2148] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:31:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:31:46.567165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1079/2149] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:46 2022] DELETE /identity/v3/users/4c70e54e4b3b407295a2d0c9ecb459bc => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:46.597827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c4f21d-a555-4704-97d7-fb0eb05acb6d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:46.598127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c4f21d-a555-4704-97d7-fb0eb05acb6d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:46.598904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c4f21d-a555-4704-97d7-fb0eb05acb6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:46.624963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c8c4f21d-a555-4704-97d7-fb0eb05acb6d None None] MFA Rules not processed for user `4bc6be7644b24ca9bab23128b41a7148`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:46.626251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c8c4f21d-a555-4704-97d7-fb0eb05acb6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:46.741354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:46.743158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:46.771519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1071/2150] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:31:46 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:46.787752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:46.811520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed8396-9ba4-41f3-bf53-5371ab0cf45c None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:46.812001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed8396-9ba4-41f3-bf53-5371ab0cf45c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:46.812400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed8396-9ba4-41f3-bf53-5371ab0cf45c None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:46.813967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1072/2151] 10.222.0.16 () {54 vars in 961 bytes} [Tue Sep 13 15:31:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:31:46.822267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb44768-434e-465c-a1a0-587592148c8e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:46.822637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb44768-434e-465c-a1a0-587592148c8e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:46.822849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb44768-434e-465c-a1a0-587592148c8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:46.878602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-3eb44768-434e-465c-a1a0-587592148c8e None None] MFA Rules not processed for user `4bc6be7644b24ca9bab23128b41a7148`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:46.882573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3eb44768-434e-465c-a1a0-587592148c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:46.887254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b580e9-7df8-4c38-8796-4055ff586dc8 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:46.895665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1073/2152] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Sep 13 15:31:46 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:46.898824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:46.900023 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:46.900373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:46.900709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] PATH_INFO: `/v3/users/c8dcc087badb43b7a18d294a6e316fe0` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:46.902070 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8dcc087badb43b7a18d294a6e316fe0)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:46.904619 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:46.907779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb44768-434e-465c-a1a0-587592148c8e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:46.919409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:46.928590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb44768-434e-465c-a1a0-587592148c8e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:46.944220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b360a97a-c03a-485f-a1f1-c016e31eff74 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:46.945273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b360a97a-c03a-485f-a1f1-c016e31eff74 service placement] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:46.945649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b360a97a-c03a-485f-a1f1-c016e31eff74 service placement] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:46.946025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b360a97a-c03a-485f-a1f1-c016e31eff74 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:46.956662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b360a97a-c03a-485f-a1f1-c016e31eff74 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:46.958833 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b360a97a-c03a-485f-a1f1-c016e31eff74 service placement] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:46.971298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1074/2153] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:47.021447 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8dcc087badb43b7a18d294a6e316fe0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.022214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8dcc087badb43b7a18d294a6e316fe0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.032442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b360a97a-c03a-485f-a1f1-c016e31eff74 service placement] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.035086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b360a97a-c03a-485f-a1f1-c016e31eff74 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:47.038163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8dcc087badb43b7a18d294a6e316fe0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.041037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8dcc087badb43b7a18d294a6e316fe0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.063114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb9c129b-8a46-4623-b524-919cd0a11ab1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8dcc087badb43b7a18d294a6e316fe0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.067525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b360a97a-c03a-485f-a1f1-c016e31eff74 service placement] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.079628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.080518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.081114 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.081458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] PATH_INFO: `/v3/projects/90980e4d215d43a5924a03aec1268de7` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.083451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=90980e4d215d43a5924a03aec1268de7)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.085518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1080/2154] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:46 2022] DELETE /identity/v3/users/c8dcc087badb43b7a18d294a6e316fe0 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:47.085869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.146901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90980e4d215d43a5924a03aec1268de7', 'request_id': 'req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.152799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90980e4d215d43a5924a03aec1268de7', 'request_id': 'req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.163980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90980e4d215d43a5924a03aec1268de7', 'request_id': 'req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.178332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] The token cache is being invalidate because project 90980e4d215d43a5924a03aec1268de7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:47.178675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.179282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.181086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1075/2155] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:47 2022] DELETE /identity/v3/projects/90980e4d215d43a5924a03aec1268de7 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:47.185788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e76b6-2c1b-456a-a0f8-aeae06c2f46c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.186073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e76b6-2c1b-456a-a0f8-aeae06c2f46c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.186372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e76b6-2c1b-456a-a0f8-aeae06c2f46c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.210290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-5d6e76b6-2c1b-456a-a0f8-aeae06c2f46c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:47.211698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-5d6e76b6-2c1b-456a-a0f8-aeae06c2f46c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:47.275545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.278096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:47.281834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1081/2156] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:47.292265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bcf51b-da1b-4a55-a62c-b13d5012f7ea None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.292498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bcf51b-da1b-4a55-a62c-b13d5012f7ea None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.292881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bcf51b-da1b-4a55-a62c-b13d5012f7ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.306498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40538c3b-7cf8-41dd-9911-bc07dd16dbe4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.313850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.315665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.316092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.316371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] PATH_INFO: `/v3/projects/836a885e5c97498db1721adb1a22a05f` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.317692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=836a885e5c97498db1721adb1a22a05f)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.320073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.321121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e9bcf51b-da1b-4a55-a62c-b13d5012f7ea None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:47.322809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e9bcf51b-da1b-4a55-a62c-b13d5012f7ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:47.332158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1082/2157] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:47.335954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bcf51b-da1b-4a55-a62c-b13d5012f7ea None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.347559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bcf51b-da1b-4a55-a62c-b13d5012f7ea None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.352604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3921f1e7-3cc0-4c5f-b17a-82924be9f1ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.353281 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3921f1e7-3cc0-4c5f-b17a-82924be9f1ec admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.353617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3921f1e7-3cc0-4c5f-b17a-82924be9f1ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.354010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3921f1e7-3cc0-4c5f-b17a-82924be9f1ec admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.354805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3921f1e7-3cc0-4c5f-b17a-82924be9f1ec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:47.355226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3921f1e7-3cc0-4c5f-b17a-82924be9f1ec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.357279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3921f1e7-3cc0-4c5f-b17a-82924be9f1ec admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.374744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1083/2158] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:31:47.375608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3921f1e7-3cc0-4c5f-b17a-82924be9f1ec admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.384094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '836a885e5c97498db1721adb1a22a05f', 'request_id': 'req-003e2043-dbd7-435c-99d1-8dfcd72c4c97'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.385477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3921f1e7-3cc0-4c5f-b17a-82924be9f1ec admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.391713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '836a885e5c97498db1721adb1a22a05f', 'request_id': 'req-003e2043-dbd7-435c-99d1-8dfcd72c4c97'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.400647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '836a885e5c97498db1721adb1a22a05f', 'request_id': 'req-003e2043-dbd7-435c-99d1-8dfcd72c4c97'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.410204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:47.413069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] The token cache is being invalidate because project 836a885e5c97498db1721adb1a22a05f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:47.413322 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.414002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-003e2043-dbd7-435c-99d1-8dfcd72c4c97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:47.415546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1076/2159] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:47 2022] DELETE /identity/v3/projects/836a885e5c97498db1721adb1a22a05f => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:47.420564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dd4d44-b5d2-4b5f-b5d9-54fd0917294d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.420850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dd4d44-b5d2-4b5f-b5d9-54fd0917294d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.421124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dd4d44-b5d2-4b5f-b5d9-54fd0917294d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.442128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a0dd4d44-b5d2-4b5f-b5d9-54fd0917294d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:47.443363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a0dd4d44-b5d2-4b5f-b5d9-54fd0917294d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:47.446762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9d59eb-e652-4520-8d54-39298eaec5fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.447503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9d59eb-e652-4520-8d54-39298eaec5fa admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.447788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9d59eb-e652-4520-8d54-39298eaec5fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.448061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9d59eb-e652-4520-8d54-39298eaec5fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.448832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9d59eb-e652-4520-8d54-39298eaec5fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.450839 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9d59eb-e652-4520-8d54-39298eaec5fa admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.451480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1077/2160] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:47.459665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1808a64e-7f8a-482f-969b-b06deb475993 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.459952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1808a64e-7f8a-482f-969b-b06deb475993 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.460246 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1808a64e-7f8a-482f-969b-b06deb475993 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.484221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1084/2161] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:47 2022] POST /identity/v3/projects => generated 367 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:47.484417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-1808a64e-7f8a-482f-969b-b06deb475993 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:47.484417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1808a64e-7f8a-482f-969b-b06deb475993 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:47.485282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9d59eb-e652-4520-8d54-39298eaec5fa admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.487018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ab9d59eb-e652-4520-8d54-39298eaec5fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:47.530424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:47.546445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1078/2162] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:47.550636 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1808a64e-7f8a-482f-969b-b06deb475993 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.552125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9d59eb-e652-4520-8d54-39298eaec5fa admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.557422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77f6738-c557-49b0-a46c-21ba57a74880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.558074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77f6738-c557-49b0-a46c-21ba57a74880 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.558372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77f6738-c557-49b0-a46c-21ba57a74880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.558648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77f6738-c557-49b0-a46c-21ba57a74880 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.559287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77f6738-c557-49b0-a46c-21ba57a74880 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.559870 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1808a64e-7f8a-482f-969b-b06deb475993 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.561237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77f6738-c557-49b0-a46c-21ba57a74880 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.564639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a5f26b-2b81-4edc-a384-8396202ec13f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.565283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a5f26b-2b81-4edc-a384-8396202ec13f admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.565544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a5f26b-2b81-4edc-a384-8396202ec13f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.565820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a5f26b-2b81-4edc-a384-8396202ec13f admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.566569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a5f26b-2b81-4edc-a384-8396202ec13f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:47.566868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a5f26b-2b81-4edc-a384-8396202ec13f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.568525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a5f26b-2b81-4edc-a384-8396202ec13f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.582043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1079/2163] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:47.585441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1085/2164] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:47 2022] POST /identity/v3/users => generated 284 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:31:47.589330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a5f26b-2b81-4edc-a384-8396202ec13f admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.589476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77f6738-c557-49b0-a46c-21ba57a74880 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.599879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a5f26b-2b81-4edc-a384-8396202ec13f admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.600665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77f6738-c557-49b0-a46c-21ba57a74880 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.604750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eab6cc8-25e6-4ac8-a4a7-d16b1d62a8ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.605347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eab6cc8-25e6-4ac8-a4a7-d16b1d62a8ee admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.605620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eab6cc8-25e6-4ac8-a4a7-d16b1d62a8ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.605885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eab6cc8-25e6-4ac8-a4a7-d16b1d62a8ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.606156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d559568-9a5c-4a49-8dd7-06062d9e704d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.606603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eab6cc8-25e6-4ac8-a4a7-d16b1d62a8ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.606763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d559568-9a5c-4a49-8dd7-06062d9e704d admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.607037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d559568-9a5c-4a49-8dd7-06062d9e704d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.607307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d559568-9a5c-4a49-8dd7-06062d9e704d admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.608022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d559568-9a5c-4a49-8dd7-06062d9e704d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.608489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eab6cc8-25e6-4ac8-a4a7-d16b1d62a8ee admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.609545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d559568-9a5c-4a49-8dd7-06062d9e704d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.619729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1086/2165] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:47.623862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d559568-9a5c-4a49-8dd7-06062d9e704d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.637780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d559568-9a5c-4a49-8dd7-06062d9e704d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.638352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1080/2166] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:47 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:31:47.643663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eab6cc8-25e6-4ac8-a4a7-d16b1d62a8ee admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.698853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9dd2f8-903c-406a-bfca-a53b64f5b81d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.699679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9dd2f8-903c-406a-bfca-a53b64f5b81d admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.700026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9dd2f8-903c-406a-bfca-a53b64f5b81d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.701135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9dd2f8-903c-406a-bfca-a53b64f5b81d admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.701135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9dd2f8-903c-406a-bfca-a53b64f5b81d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.702641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9dd2f8-903c-406a-bfca-a53b64f5b81d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.714388 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1087/2167] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:47 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:47.718196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9dd2f8-903c-406a-bfca-a53b64f5b81d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.722746 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eab6cc8-25e6-4ac8-a4a7-d16b1d62a8ee admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.730005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9dd2f8-903c-406a-bfca-a53b64f5b81d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.730291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fec7a5-1703-4309-9c92-09479a2748a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.730946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fec7a5-1703-4309-9c92-09479a2748a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.731237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fec7a5-1703-4309-9c92-09479a2748a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.731505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fec7a5-1703-4309-9c92-09479a2748a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.732141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fec7a5-1703-4309-9c92-09479a2748a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.734185 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fec7a5-1703-4309-9c92-09479a2748a6 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.735152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6c5896-3c85-4139-a583-7791bdc683e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.735908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6c5896-3c85-4139-a583-7791bdc683e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.736277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6c5896-3c85-4139-a583-7791bdc683e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.736706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6c5896-3c85-4139-a583-7791bdc683e6 admin admin] PATH_INFO: `/v3/projects/73fb66ba3f5942efb8720c416991a6fc/users/97437e4800e249249316bdf988b6e42a/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.753482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6c5896-3c85-4139-a583-7791bdc683e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73fb66ba3f5942efb8720c416991a6fc, user_id=97437e4800e249249316bdf988b6e42a, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.756488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6c5896-3c85-4139-a583-7791bdc683e6 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.760569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1081/2168] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:47 2022] POST /identity/v3/users => generated 293 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:47.766019 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fec7a5-1703-4309-9c92-09479a2748a6 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.773047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1088/2169] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:47 2022] PUT /identity/v3/projects/73fb66ba3f5942efb8720c416991a6fc/users/97437e4800e249249316bdf988b6e42a/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:47.778141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fec7a5-1703-4309-9c92-09479a2748a6 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.784054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703edf5-ce9d-4459-8b53-c7eaad673b24 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.784472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703edf5-ce9d-4459-8b53-c7eaad673b24 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.784810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703edf5-ce9d-4459-8b53-c7eaad673b24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.792439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:47.821822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7703edf5-ce9d-4459-8b53-c7eaad673b24 None None] MFA Rules not processed for user `97437e4800e249249316bdf988b6e42a`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:47.823206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7703edf5-ce9d-4459-8b53-c7eaad673b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:47.852937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a914c5-0a67-44bb-b31d-e5186f424c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.852937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a914c5-0a67-44bb-b31d-e5186f424c12 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.853285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a914c5-0a67-44bb-b31d-e5186f424c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.853826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a914c5-0a67-44bb-b31d-e5186f424c12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.854130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a914c5-0a67-44bb-b31d-e5186f424c12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.855892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a914c5-0a67-44bb-b31d-e5186f424c12 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.870025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1082/2170] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:47 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:47.871515 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a914c5-0a67-44bb-b31d-e5186f424c12 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.881781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a914c5-0a67-44bb-b31d-e5186f424c12 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.886620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f543b-15b3-47f4-816a-aeb237bf6879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.887385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f543b-15b3-47f4-816a-aeb237bf6879 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.887662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f543b-15b3-47f4-816a-aeb237bf6879 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.887967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96f543b-15b3-47f4-816a-aeb237bf6879 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.888701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f543b-15b3-47f4-816a-aeb237bf6879 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.890556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96f543b-15b3-47f4-816a-aeb237bf6879 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.901376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1083/2171] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:47.905237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f543b-15b3-47f4-816a-aeb237bf6879 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.905920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1089/2172] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:47 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:47.916542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96f543b-15b3-47f4-816a-aeb237bf6879 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:47.917770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7703edf5-ce9d-4459-8b53-c7eaad673b24 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:47.919333 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7703edf5-ce9d-4459-8b53-c7eaad673b24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:47.922059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f406c712-1d20-4245-8dec-0d449b298ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:47.922814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406c712-1d20-4245-8dec-0d449b298ba4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.923155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406c712-1d20-4245-8dec-0d449b298ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.923489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406c712-1d20-4245-8dec-0d449b298ba4 admin admin] PATH_INFO: `/v3/projects/76163bf55e0a403aaec08c9c6a68a55f/users/f48ce3ac32e741ef98e5025fd23bbaa0/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.943333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f406c712-1d20-4245-8dec-0d449b298ba4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76163bf55e0a403aaec08c9c6a68a55f, user_id=f48ce3ac32e741ef98e5025fd23bbaa0, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:47.958129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f406c712-1d20-4245-8dec-0d449b298ba4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:47.966453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1084/2173] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:47 2022] PUT /identity/v3/projects/76163bf55e0a403aaec08c9c6a68a55f/users/f48ce3ac32e741ef98e5025fd23bbaa0/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:47.968461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9379c743-eca1-4fec-a8df-ac416ae6557a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:47.968966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9379c743-eca1-4fec-a8df-ac416ae6557a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:47.969357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9379c743-eca1-4fec-a8df-ac416ae6557a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:47.985128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:47.993332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-9379c743-eca1-4fec-a8df-ac416ae6557a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:47.996721 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9379c743-eca1-4fec-a8df-ac416ae6557a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:48.030824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7703edf5-ce9d-4459-8b53-c7eaad673b24 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:48.068448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1085/2174] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:48.081173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9379c743-eca1-4fec-a8df-ac416ae6557a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:48.086249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9379c743-eca1-4fec-a8df-ac416ae6557a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:48.099870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af28c9f-7b36-42ff-8f0e-14538497e16e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce646dcb8124685ac7a1c1eca664d04', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:48.100945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af28c9f-7b36-42ff-8f0e-14538497e16e service glance] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:48.101432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af28c9f-7b36-42ff-8f0e-14538497e16e service glance] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:48.101843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af28c9f-7b36-42ff-8f0e-14538497e16e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:48.148259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:48.148517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9379c743-eca1-4fec-a8df-ac416ae6557a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:48.152937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4989e266-26e4-427a-ac51-b30431d0b6e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:48.153745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4989e266-26e4-427a-ac51-b30431d0b6e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:48.154018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4989e266-26e4-427a-ac51-b30431d0b6e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:48.154366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4989e266-26e4-427a-ac51-b30431d0b6e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:48.163817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4989e266-26e4-427a-ac51-b30431d0b6e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:48.165256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4989e266-26e4-427a-ac51-b30431d0b6e5 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:48.165852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af28c9f-7b36-42ff-8f0e-14538497e16e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:48.167410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af28c9f-7b36-42ff-8f0e-14538497e16e service glance] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:48.175617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1086/2175] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:31:48.176799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1090/2176] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:47 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:53.604801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4989e266-26e4-427a-ac51-b30431d0b6e5 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:53.607544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4989e266-26e4-427a-ac51-b30431d0b6e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:53.643791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4989e266-26e4-427a-ac51-b30431d0b6e5 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:53.674333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:53.675115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:53.675396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:53.675662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] PATH_INFO: `/v3/users/b05ed8547e514d858ca26cc7708285a6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:53.676905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] RBAC: Authorizing `identity:delete_user(user_id=b05ed8547e514d858ca26cc7708285a6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:53.684955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:53.782885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b05ed8547e514d858ca26cc7708285a6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:53.793429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b05ed8547e514d858ca26cc7708285a6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:53.794243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b05ed8547e514d858ca26cc7708285a6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:53.794947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b05ed8547e514d858ca26cc7708285a6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:53.804392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b05ed8547e514d858ca26cc7708285a6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:53.829992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1087/2177] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:31:53 2022] DELETE /identity/v3/users/b05ed8547e514d858ca26cc7708285a6 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:53.834496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef63a50-0a66-477e-9b7d-ac3aa2d7ab00 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:53.835050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef63a50-0a66-477e-9b7d-ac3aa2d7ab00 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:53.835393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef63a50-0a66-477e-9b7d-ac3aa2d7ab00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:53.863926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-fef63a50-0a66-477e-9b7d-ac3aa2d7ab00 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:53.866821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fef63a50-0a66-477e-9b7d-ac3aa2d7ab00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:53.956358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1091/2178] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:53.974668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:53.976384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:54.016014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:54.088084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8d1021-b3c5-4ee0-af5f-a9b70d30320f admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:54.100549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5e6e06-c8ad-4f76-8dc0-d5ab16bf94c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:54.101638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5e6e06-c8ad-4f76-8dc0-d5ab16bf94c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:54.102738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5e6e06-c8ad-4f76-8dc0-d5ab16bf94c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:54.103187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5e6e06-c8ad-4f76-8dc0-d5ab16bf94c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:54.118455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5e6e06-c8ad-4f76-8dc0-d5ab16bf94c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:54.120218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5e6e06-c8ad-4f76-8dc0-d5ab16bf94c7 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:54.146240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1088/2179] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:54.557002 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef63a50-0a66-477e-9b7d-ac3aa2d7ab00 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:54.558574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fef63a50-0a66-477e-9b7d-ac3aa2d7ab00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:54.573416 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef63a50-0a66-477e-9b7d-ac3aa2d7ab00 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:54.585710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:54.586520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:54.586867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:54.587212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] PATH_INFO: `/v3/projects/c4ea71ef4ef9490eb90e3405499df93d` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:54.593372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4ea71ef4ef9490eb90e3405499df93d)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:54.597244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:54.700309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4ea71ef4ef9490eb90e3405499df93d', 'request_id': 'req-b6294790-ae35-462b-825c-3a34368710a0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:54.723238 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4ea71ef4ef9490eb90e3405499df93d', 'request_id': 'req-b6294790-ae35-462b-825c-3a34368710a0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:54.749972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4ea71ef4ef9490eb90e3405499df93d', 'request_id': 'req-b6294790-ae35-462b-825c-3a34368710a0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:54.770754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] The token cache is being invalidate because project c4ea71ef4ef9490eb90e3405499df93d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:31:54.771052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:54.771815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b6294790-ae35-462b-825c-3a34368710a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:31:54.778189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1092/2180] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:31:54 2022] DELETE /identity/v3/projects/c4ea71ef4ef9490eb90e3405499df93d => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:31:54.780795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03dba08-03ed-43ff-a2d4-f29098ba6050 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:54.781287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03dba08-03ed-43ff-a2d4-f29098ba6050 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:54.781397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03dba08-03ed-43ff-a2d4-f29098ba6050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:54.827162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e03dba08-03ed-43ff-a2d4-f29098ba6050 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:54.828382 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e03dba08-03ed-43ff-a2d4-f29098ba6050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:54.909137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1089/2181] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:54.917319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92c212b-ebfc-4e54-9ac9-d61b9b010367 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:54.917826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92c212b-ebfc-4e54-9ac9-d61b9b010367 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:54.917939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92c212b-ebfc-4e54-9ac9-d61b9b010367 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:54.945342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c92c212b-ebfc-4e54-9ac9-d61b9b010367 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:54.946725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c92c212b-ebfc-4e54-9ac9-d61b9b010367 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:54.959087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1093/2182] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:54.963516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03dba08-03ed-43ff-a2d4-f29098ba6050 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:54.978710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03dba08-03ed-43ff-a2d4-f29098ba6050 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:54.999883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e30ef2-d136-4ead-bb65-48ae0843506c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:55.000870 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e30ef2-d136-4ead-bb65-48ae0843506c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:55.000870 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e30ef2-d136-4ead-bb65-48ae0843506c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:55.001097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e30ef2-d136-4ead-bb65-48ae0843506c admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:55.001735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e30ef2-d136-4ead-bb65-48ae0843506c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:31:55.002075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e30ef2-d136-4ead-bb65-48ae0843506c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:55.003932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e30ef2-d136-4ead-bb65-48ae0843506c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:55.027134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1090/2183] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:31:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:31:55.029092 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92c212b-ebfc-4e54-9ac9-d61b9b010367 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:55.049639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92c212b-ebfc-4e54-9ac9-d61b9b010367 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:55.058491 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce53253-cedc-4f73-bd88-b51977b4ccd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:55.059543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce53253-cedc-4f73-bd88-b51977b4ccd1 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:55.059853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce53253-cedc-4f73-bd88-b51977b4ccd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:55.060182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce53253-cedc-4f73-bd88-b51977b4ccd1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:55.061029 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce53253-cedc-4f73-bd88-b51977b4ccd1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:55.063342 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce53253-cedc-4f73-bd88-b51977b4ccd1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:55.150710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1094/2184] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:31:55 2022] POST /identity/v3/projects => generated 379 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:31:55.153807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e30ef2-d136-4ead-bb65-48ae0843506c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:55.170026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e30ef2-d136-4ead-bb65-48ae0843506c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:55.243979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9821447b-4fd2-413e-9d63-eb5ccecc5430 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:55.244335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9821447b-4fd2-413e-9d63-eb5ccecc5430 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:55.244676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9821447b-4fd2-413e-9d63-eb5ccecc5430 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:55.255434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e2705d-573b-4ce5-964a-5dc070636c23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:55.256045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e2705d-573b-4ce5-964a-5dc070636c23 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:55.256316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e2705d-573b-4ce5-964a-5dc070636c23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:55.256577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e2705d-573b-4ce5-964a-5dc070636c23 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:55.257214 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e2705d-573b-4ce5-964a-5dc070636c23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:55.259235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e2705d-573b-4ce5-964a-5dc070636c23 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:55.275706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-9821447b-4fd2-413e-9d63-eb5ccecc5430 None None] MFA Rules not processed for user `f48ce3ac32e741ef98e5025fd23bbaa0`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:55.278766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9821447b-4fd2-413e-9d63-eb5ccecc5430 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:55.286055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1091/2185] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:31:55 2022] POST /identity/v3/users => generated 290 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:31:55.290247 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e2705d-573b-4ce5-964a-5dc070636c23 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:55.304865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e2705d-573b-4ce5-964a-5dc070636c23 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:55.311224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3772ed-298e-474f-81ab-9bfa5ebd4f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:55.311851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3772ed-298e-474f-81ab-9bfa5ebd4f09 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:55.312124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3772ed-298e-474f-81ab-9bfa5ebd4f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:55.312694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3772ed-298e-474f-81ab-9bfa5ebd4f09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:55.313187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3772ed-298e-474f-81ab-9bfa5ebd4f09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:55.315544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3772ed-298e-474f-81ab-9bfa5ebd4f09 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:55.319972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:55.335309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1092/2186] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:55 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:31:55.340226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3772ed-298e-474f-81ab-9bfa5ebd4f09 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:55.354705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3772ed-298e-474f-81ab-9bfa5ebd4f09 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:55.359333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ed8d70-d0fb-4e8b-b356-eb828fc55800 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:55.360491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ed8d70-d0fb-4e8b-b356-eb828fc55800 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:55.361155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ed8d70-d0fb-4e8b-b356-eb828fc55800 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:55.362348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ed8d70-d0fb-4e8b-b356-eb828fc55800 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:55.364155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ed8d70-d0fb-4e8b-b356-eb828fc55800 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:55.365869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ed8d70-d0fb-4e8b-b356-eb828fc55800 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:55.385194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1093/2187] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:31:55 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:31:55.385568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1095/2188] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:55 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:31:55.386409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ed8d70-d0fb-4e8b-b356-eb828fc55800 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:55.400364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ed8d70-d0fb-4e8b-b356-eb828fc55800 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:55.403911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9821447b-4fd2-413e-9d63-eb5ccecc5430 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:55.405357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9821447b-4fd2-413e-9d63-eb5ccecc5430 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:55.406378 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e448370-da57-4d69-a941-0b3d7ee85934 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:55.407120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e448370-da57-4d69-a941-0b3d7ee85934 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:55.407394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e448370-da57-4d69-a941-0b3d7ee85934 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:55.407687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e448370-da57-4d69-a941-0b3d7ee85934 admin admin] PATH_INFO: `/v3/projects/f99e1e9c3c13491e86734af2b526fc42/users/1dbbc500cd8c4cedb3784d7750cb6de2/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:55.429947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e448370-da57-4d69-a941-0b3d7ee85934 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f99e1e9c3c13491e86734af2b526fc42, user_id=1dbbc500cd8c4cedb3784d7750cb6de2, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:55.434056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e448370-da57-4d69-a941-0b3d7ee85934 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:55.437786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:31:55.454926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1094/2189] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:31:55 2022] PUT /identity/v3/projects/f99e1e9c3c13491e86734af2b526fc42/users/1dbbc500cd8c4cedb3784d7750cb6de2/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:31:55.460641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e2ff89-073b-41b2-8851-37bd2a744a83 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:55.461644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e2ff89-073b-41b2-8851-37bd2a744a83 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:55.461924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e2ff89-073b-41b2-8851-37bd2a744a83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:55.483357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b3e2ff89-073b-41b2-8851-37bd2a744a83 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:55.484615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b3e2ff89-073b-41b2-8851-37bd2a744a83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:55.510603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9821447b-4fd2-413e-9d63-eb5ccecc5430 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:55.562154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1095/2190] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:31:55.576636 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e2ff89-073b-41b2-8851-37bd2a744a83 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:31:55.586880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b3e2ff89-073b-41b2-8851-37bd2a744a83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:55.617957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453189e7-496f-4419-b3cb-35128adfed2b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:55.618912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453189e7-496f-4419-b3cb-35128adfed2b service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:55.619193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453189e7-496f-4419-b3cb-35128adfed2b service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:55.619466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453189e7-496f-4419-b3cb-35128adfed2b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:55.661477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e2ff89-073b-41b2-8851-37bd2a744a83 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:31:55.665365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f7a681-1f5e-406f-ada8-1ed5034dc40b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:31:55.665986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f7a681-1f5e-406f-ada8-1ed5034dc40b service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:55.666301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f7a681-1f5e-406f-ada8-1ed5034dc40b service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:55.666568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f7a681-1f5e-406f-ada8-1ed5034dc40b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:55.680257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f7a681-1f5e-406f-ada8-1ed5034dc40b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:55.681732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f7a681-1f5e-406f-ada8-1ed5034dc40b service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:55.693038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1096/2191] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:31:55.696795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453189e7-496f-4419-b3cb-35128adfed2b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:31:55.698467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453189e7-496f-4419-b3cb-35128adfed2b service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:31:55.710628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1096/2192] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:31:55 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:31:57.930674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870981a5-529f-4f03-9f18-3a3ae2fadd6e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:31:57.931257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870981a5-529f-4f03-9f18-3a3ae2fadd6e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:31:57.931257 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870981a5-529f-4f03-9f18-3a3ae2fadd6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:31:57.974163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-870981a5-529f-4f03-9f18-3a3ae2fadd6e None None] MFA Rules not processed for user `f48ce3ac32e741ef98e5025fd23bbaa0`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:31:57.975549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-870981a5-529f-4f03-9f18-3a3ae2fadd6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:31:58.006285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1097/2193] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:31:57 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:02.262586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e9fe79-02b2-4f79-9ed8-1fa45671a691 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:02.262586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e9fe79-02b2-4f79-9ed8-1fa45671a691 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:02.262586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e9fe79-02b2-4f79-9ed8-1fa45671a691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:02.275071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:02.291706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-74e9fe79-02b2-4f79-9ed8-1fa45671a691 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:02.293445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-74e9fe79-02b2-4f79-9ed8-1fa45671a691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:02.362836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1097/2194] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:32:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:02.410205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870981a5-529f-4f03-9f18-3a3ae2fadd6e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:02.412244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-870981a5-529f-4f03-9f18-3a3ae2fadd6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:02.606027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870981a5-529f-4f03-9f18-3a3ae2fadd6e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:02.613472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dfbfa9-d01a-4650-bc80-f43b6e4772e4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:02.623958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dfbfa9-d01a-4650-bc80-f43b6e4772e4 service swift] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:02.630676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dfbfa9-d01a-4650-bc80-f43b6e4772e4 service swift] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:02.631199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dfbfa9-d01a-4650-bc80-f43b6e4772e4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:02.663081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dfbfa9-d01a-4650-bc80-f43b6e4772e4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:02.669276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dfbfa9-d01a-4650-bc80-f43b6e4772e4 service swift] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:02.703465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1098/2195] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:32:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:32:03.627765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1f94e8-7767-43ce-9403-1bbb83407226 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:03.628587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1f94e8-7767-43ce-9403-1bbb83407226 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:03.629145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1f94e8-7767-43ce-9403-1bbb83407226 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:03.676171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2f1f94e8-7767-43ce-9403-1bbb83407226 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:03.677852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2f1f94e8-7767-43ce-9403-1bbb83407226 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:03.699679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1098/2196] 10.222.0.16 () {58 vars in 967 bytes} [Tue Sep 13 15:32:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:32:03.714853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dfbfa9-d01a-4650-bc80-f43b6e4772e4 service swift] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:03.728638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dfbfa9-d01a-4650-bc80-f43b6e4772e4 service swift] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:03.743399 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b67141f-6b6c-4d0f-a75f-2c99b934c414 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:03.744144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b67141f-6b6c-4d0f-a75f-2c99b934c414 service swift] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:03.744547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b67141f-6b6c-4d0f-a75f-2c99b934c414 service swift] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:03.744930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b67141f-6b6c-4d0f-a75f-2c99b934c414 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:03.781866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b67141f-6b6c-4d0f-a75f-2c99b934c414 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:03.783962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b67141f-6b6c-4d0f-a75f-2c99b934c414 service swift] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:03.798426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1099/2197] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:32:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 13 15:32:04.217981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abc9439-4adb-4d5d-9085-7bfd63b63f0a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:04.218432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abc9439-4adb-4d5d-9085-7bfd63b63f0a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:04.218935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abc9439-4adb-4d5d-9085-7bfd63b63f0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:04.275711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2abc9439-4adb-4d5d-9085-7bfd63b63f0a None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:04.277428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2abc9439-4adb-4d5d-9085-7bfd63b63f0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:04.285976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1099/2198] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:32:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:04.301998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b67141f-6b6c-4d0f-a75f-2c99b934c414 service swift] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:04.313307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b67141f-6b6c-4d0f-a75f-2c99b934c414 service swift] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:04.329858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b078d06-f3f0-4894-aeee-592516472095 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:04.330528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b078d06-f3f0-4894-aeee-592516472095 service swift] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:04.330800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b078d06-f3f0-4894-aeee-592516472095 service swift] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:04.331128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b078d06-f3f0-4894-aeee-592516472095 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:04.340842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b078d06-f3f0-4894-aeee-592516472095 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:04.342813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b078d06-f3f0-4894-aeee-592516472095 service swift] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:04.348237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:04.355517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1100/2199] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:32:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:32:04.916873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52510b1-83b0-4d01-a80c-b6e94aceba06 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:04.917360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52510b1-83b0-4d01-a80c-b6e94aceba06 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:04.917488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52510b1-83b0-4d01-a80c-b6e94aceba06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:04.950438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d52510b1-83b0-4d01-a80c-b6e94aceba06 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:04.950438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d52510b1-83b0-4d01-a80c-b6e94aceba06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:04.955728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1100/2200] 10.222.0.16 () {58 vars in 967 bytes} [Tue Sep 13 15:32:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:04.971008 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b078d06-f3f0-4894-aeee-592516472095 service swift] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:04.981589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b078d06-f3f0-4894-aeee-592516472095 service swift] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:04.987265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b6d84-2bba-454d-b47f-7416a76e3b0a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:04.988011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b6d84-2bba-454d-b47f-7416a76e3b0a service swift] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:04.988504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b6d84-2bba-454d-b47f-7416a76e3b0a service swift] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:04.988990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b6d84-2bba-454d-b47f-7416a76e3b0a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:04.999234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014b6d84-2bba-454d-b47f-7416a76e3b0a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:05.000943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014b6d84-2bba-454d-b47f-7416a76e3b0a service swift] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:05.014044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1101/2201] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:32:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:32:05.026154 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2c40d7-f8b0-4b01-902c-3136f1c8583f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:05.026588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2c40d7-f8b0-4b01-902c-3136f1c8583f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:05.027588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2c40d7-f8b0-4b01-902c-3136f1c8583f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:05.086191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-fa2c40d7-f8b0-4b01-902c-3136f1c8583f None None] MFA Rules not processed for user `1dbbc500cd8c4cedb3784d7750cb6de2`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:05.088501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fa2c40d7-f8b0-4b01-902c-3136f1c8583f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:05.196536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1101/2202] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:05 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:05.197832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b6d84-2bba-454d-b47f-7416a76e3b0a service swift] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:05.208723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b6d84-2bba-454d-b47f-7416a76e3b0a service swift] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:05.213455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9bf6b4-a902-45a3-b77f-e7ae2924bdf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:05.214062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9bf6b4-a902-45a3-b77f-e7ae2924bdf8 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:05.214392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9bf6b4-a902-45a3-b77f-e7ae2924bdf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:05.214656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9bf6b4-a902-45a3-b77f-e7ae2924bdf8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:05.215287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9bf6b4-a902-45a3-b77f-e7ae2924bdf8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:05.217158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9bf6b4-a902-45a3-b77f-e7ae2924bdf8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:05.261495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1102/2203] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:32:05 2022] POST /identity/v3/projects => generated 375 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:32:05.262349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2c40d7-f8b0-4b01-902c-3136f1c8583f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:05.285707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2c40d7-f8b0-4b01-902c-3136f1c8583f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:05.393605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e306370-d91a-46f1-9f91-a1a796dc3771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:05.394027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e306370-d91a-46f1-9f91-a1a796dc3771 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:05.394471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e306370-d91a-46f1-9f91-a1a796dc3771 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:05.394796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e306370-d91a-46f1-9f91-a1a796dc3771 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:05.395891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e306370-d91a-46f1-9f91-a1a796dc3771 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:05.398000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e306370-d91a-46f1-9f91-a1a796dc3771 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:05.456770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1102/2204] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:32:05 2022] POST /identity/v3/users => generated 294 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:32:05.463672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9bf6b4-a902-45a3-b77f-e7ae2924bdf8 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:05.480074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9bf6b4-a902-45a3-b77f-e7ae2924bdf8 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:05.489609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224df52a-a623-4538-8b17-b1ffa605605e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:05.490295 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224df52a-a623-4538-8b17-b1ffa605605e admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:05.490573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224df52a-a623-4538-8b17-b1ffa605605e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:05.490916 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224df52a-a623-4538-8b17-b1ffa605605e admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:05.491579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224df52a-a623-4538-8b17-b1ffa605605e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:05.493277 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224df52a-a623-4538-8b17-b1ffa605605e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:05.511338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e306370-d91a-46f1-9f91-a1a796dc3771 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:05.511698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1103/2205] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:32:05 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:32:05.526650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e306370-d91a-46f1-9f91-a1a796dc3771 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:05.534535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887d1411-0fc9-4792-9782-096ee228f147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:05.536012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887d1411-0fc9-4792-9782-096ee228f147 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:05.537044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887d1411-0fc9-4792-9782-096ee228f147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:05.537572 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887d1411-0fc9-4792-9782-096ee228f147 admin admin] PATH_INFO: `/v3/projects/dc9ec7f984554ee087775cb0d36aed7c/users/d658f847aadd411390b1d7f7378b8d83/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:05.550613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:05.595939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887d1411-0fc9-4792-9782-096ee228f147 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc9ec7f984554ee087775cb0d36aed7c, user_id=d658f847aadd411390b1d7f7378b8d83, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:05.599753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887d1411-0fc9-4792-9782-096ee228f147 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:05.621283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1103/2206] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:32:05 2022] PUT /identity/v3/projects/dc9ec7f984554ee087775cb0d36aed7c/users/d658f847aadd411390b1d7f7378b8d83/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:05.625333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224df52a-a623-4538-8b17-b1ffa605605e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:05.638609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224df52a-a623-4538-8b17-b1ffa605605e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:05.774792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dadcda3-ae60-4dc4-92f2-fc385ea3f318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:05.775605 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dadcda3-ae60-4dc4-92f2-fc385ea3f318 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:05.776126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dadcda3-ae60-4dc4-92f2-fc385ea3f318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:05.776465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dadcda3-ae60-4dc4-92f2-fc385ea3f318 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:05.778571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dadcda3-ae60-4dc4-92f2-fc385ea3f318 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:05.780225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dadcda3-ae60-4dc4-92f2-fc385ea3f318 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:05.797040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1104/2207] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:32:05 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:32:05.801917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887d1411-0fc9-4792-9782-096ee228f147 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:05.819112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887d1411-0fc9-4792-9782-096ee228f147 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:05.834823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659621f0-a296-4fb3-9e83-b9860ee89fb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:05.835708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659621f0-a296-4fb3-9e83-b9860ee89fb4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:05.836152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659621f0-a296-4fb3-9e83-b9860ee89fb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:05.836648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659621f0-a296-4fb3-9e83-b9860ee89fb4 admin admin] PATH_INFO: `/v3/projects/dc9ec7f984554ee087775cb0d36aed7c/users/d658f847aadd411390b1d7f7378b8d83/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:05.838961 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659621f0-a296-4fb3-9e83-b9860ee89fb4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc9ec7f984554ee087775cb0d36aed7c, user_id=d658f847aadd411390b1d7f7378b8d83, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:05.844461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659621f0-a296-4fb3-9e83-b9860ee89fb4 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:05.863169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1104/2208] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:32:05 2022] PUT /identity/v3/projects/dc9ec7f984554ee087775cb0d36aed7c/users/d658f847aadd411390b1d7f7378b8d83/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:07.684903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dadcda3-ae60-4dc4-92f2-fc385ea3f318 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:07.687171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0dadcda3-ae60-4dc4-92f2-fc385ea3f318 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:07.703547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dadcda3-ae60-4dc4-92f2-fc385ea3f318 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:07.786093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:07.792451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:07.793234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:07.793540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] PATH_INFO: `/v3/users/47bc6dff55e847ba93eb7bfdcf4e95b2` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:07.794887 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=47bc6dff55e847ba93eb7bfdcf4e95b2)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:07.797128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:07.878835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47bc6dff55e847ba93eb7bfdcf4e95b2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:07.880933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:07.893293 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47bc6dff55e847ba93eb7bfdcf4e95b2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:07.895043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47bc6dff55e847ba93eb7bfdcf4e95b2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:07.897192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47bc6dff55e847ba93eb7bfdcf4e95b2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:07.908928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47bc6dff55e847ba93eb7bfdcf4e95b2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:07.933189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1105/2209] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:32:07 2022] DELETE /identity/v3/users/47bc6dff55e847ba93eb7bfdcf4e95b2 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:07.937438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07ae2d9-8a5b-47c0-8744-f0ad11becfc1 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:07.937927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07ae2d9-8a5b-47c0-8744-f0ad11becfc1 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:07.938176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07ae2d9-8a5b-47c0-8744-f0ad11becfc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:07.984073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-b07ae2d9-8a5b-47c0-8744-f0ad11becfc1 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:07.985383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b07ae2d9-8a5b-47c0-8744-f0ad11becfc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:08.077424 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1105/2210] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:08.087531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:08.089030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:08.175455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c59364-786f-4e73-bba7-52206b59dfcb admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:08.184512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d7fb37-0c5a-4c6c-94b8-b52b2f82c032 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:08.185353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d7fb37-0c5a-4c6c-94b8-b52b2f82c032 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:08.185784 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d7fb37-0c5a-4c6c-94b8-b52b2f82c032 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:08.186237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d7fb37-0c5a-4c6c-94b8-b52b2f82c032 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:08.198158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d7fb37-0c5a-4c6c-94b8-b52b2f82c032 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:08.199624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d7fb37-0c5a-4c6c-94b8-b52b2f82c032 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:08.211754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1106/2211] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:32:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:32:08.417708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07ae2d9-8a5b-47c0-8744-f0ad11becfc1 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:08.419534 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b07ae2d9-8a5b-47c0-8744-f0ad11becfc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:08.433539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07ae2d9-8a5b-47c0-8744-f0ad11becfc1 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:08.440379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:08.441395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:08.441775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:08.442269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] PATH_INFO: `/v3/projects/c1b581d18eee429cb4415287c1887fa7` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:08.443687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1b581d18eee429cb4415287c1887fa7)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:08.447182 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:08.513495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1b581d18eee429cb4415287c1887fa7', 'request_id': 'req-b73078c3-cc7a-4124-b2ce-3918235ded88'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:08.520145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1b581d18eee429cb4415287c1887fa7', 'request_id': 'req-b73078c3-cc7a-4124-b2ce-3918235ded88'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:08.531116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1b581d18eee429cb4415287c1887fa7', 'request_id': 'req-b73078c3-cc7a-4124-b2ce-3918235ded88'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:08.552892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] The token cache is being invalidate because project c1b581d18eee429cb4415287c1887fa7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:32:08.553401 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:08.553947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b73078c3-cc7a-4124-b2ce-3918235ded88 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:08.555776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1106/2212] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:32:08 2022] DELETE /identity/v3/projects/c1b581d18eee429cb4415287c1887fa7 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:08.560777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7256a7-3a7f-417c-b6e1-b8824fd1b24c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:08.561062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7256a7-3a7f-417c-b6e1-b8824fd1b24c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:08.561336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7256a7-3a7f-417c-b6e1-b8824fd1b24c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:08.597888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4c7256a7-3a7f-417c-b6e1-b8824fd1b24c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:08.599236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4c7256a7-3a7f-417c-b6e1-b8824fd1b24c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:08.686638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27085b91-b721-4be4-9a65-4e91ea8653c9 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:08.686922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27085b91-b721-4be4-9a65-4e91ea8653c9 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:08.687192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27085b91-b721-4be4-9a65-4e91ea8653c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:08.691732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1107/2213] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:08.714047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-27085b91-b721-4be4-9a65-4e91ea8653c9 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:08.715434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-27085b91-b721-4be4-9a65-4e91ea8653c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:08.735549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1107/2214] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:32:08.740072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7256a7-3a7f-417c-b6e1-b8824fd1b24c None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:08.752006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7256a7-3a7f-417c-b6e1-b8824fd1b24c None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:08.763565 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d60d4f-21cf-47bc-b78f-6fd8c1192ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:08.768428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d60d4f-21cf-47bc-b78f-6fd8c1192ee2 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:08.769408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d60d4f-21cf-47bc-b78f-6fd8c1192ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:08.769889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d60d4f-21cf-47bc-b78f-6fd8c1192ee2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:08.770802 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d60d4f-21cf-47bc-b78f-6fd8c1192ee2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:32:08.771337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d60d4f-21cf-47bc-b78f-6fd8c1192ee2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:08.773201 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d60d4f-21cf-47bc-b78f-6fd8c1192ee2 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:08.805207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27085b91-b721-4be4-9a65-4e91ea8653c9 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:08.813859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1108/2215] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:32:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:32:08.817584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27085b91-b721-4be4-9a65-4e91ea8653c9 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:08.824338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df700c06-b7b8-496b-a9b9-c145cf702ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:08.824973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df700c06-b7b8-496b-a9b9-c145cf702ae2 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:08.825255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df700c06-b7b8-496b-a9b9-c145cf702ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:08.825528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df700c06-b7b8-496b-a9b9-c145cf702ae2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:08.826221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df700c06-b7b8-496b-a9b9-c145cf702ae2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:08.828187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df700c06-b7b8-496b-a9b9-c145cf702ae2 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:08.879827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1108/2216] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:32:08 2022] POST /identity/v3/projects => generated 385 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:32:08.884569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d60d4f-21cf-47bc-b78f-6fd8c1192ee2 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:08.895416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d60d4f-21cf-47bc-b78f-6fd8c1192ee2 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:08.976966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:INSERT=1,keystone:DELETE=3,keystone:SELECT=96 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:08.985488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c114f95-c012-4f2e-9607-1fd3c1e17ed8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:08.986865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c114f95-c012-4f2e-9607-1fd3c1e17ed8 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:08.987726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c114f95-c012-4f2e-9607-1fd3c1e17ed8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:08.988499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c114f95-c012-4f2e-9607-1fd3c1e17ed8 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:08.989082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c114f95-c012-4f2e-9607-1fd3c1e17ed8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:08.992471 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c114f95-c012-4f2e-9607-1fd3c1e17ed8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:09.027803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1109/2217] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:32:08 2022] POST /identity/v3/users => generated 293 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:32:09.029049 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df700c06-b7b8-496b-a9b9-c145cf702ae2 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:09.040621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df700c06-b7b8-496b-a9b9-c145cf702ae2 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:09.047158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d888fd6-882e-41bd-8a97-6eb522b48f55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:09.048376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d888fd6-882e-41bd-8a97-6eb522b48f55 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:09.048865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d888fd6-882e-41bd-8a97-6eb522b48f55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:09.049337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d888fd6-882e-41bd-8a97-6eb522b48f55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:09.051038 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d888fd6-882e-41bd-8a97-6eb522b48f55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:09.053030 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d888fd6-882e-41bd-8a97-6eb522b48f55 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:09.069774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1109/2218] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:32:09 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:32:09.071084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c114f95-c012-4f2e-9607-1fd3c1e17ed8 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:09.086385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c114f95-c012-4f2e-9607-1fd3c1e17ed8 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:09.091713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8e5999-2a62-45a3-b63c-6875aa83aced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:09.092825 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8e5999-2a62-45a3-b63c-6875aa83aced admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:09.093311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8e5999-2a62-45a3-b63c-6875aa83aced admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:09.093821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8e5999-2a62-45a3-b63c-6875aa83aced admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:09.095324 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8e5999-2a62-45a3-b63c-6875aa83aced admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:09.097160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8e5999-2a62-45a3-b63c-6875aa83aced admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:09.117094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d888fd6-882e-41bd-8a97-6eb522b48f55 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:09.128807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1110/2219] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:32:09 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:32:09.130276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d888fd6-882e-41bd-8a97-6eb522b48f55 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:09.135844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecae20a-5a7b-4da0-90a0-841cd0bae169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:09.136717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecae20a-5a7b-4da0-90a0-841cd0bae169 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:09.137179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecae20a-5a7b-4da0-90a0-841cd0bae169 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:09.137287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecae20a-5a7b-4da0-90a0-841cd0bae169 admin admin] PATH_INFO: `/v3/projects/74318a6f074f4d7ea084461473ef1a80/users/0815b4e3389240ada8d1c55941aca8cf/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:09.150673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:09.153974 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecae20a-5a7b-4da0-90a0-841cd0bae169 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74318a6f074f4d7ea084461473ef1a80, user_id=0815b4e3389240ada8d1c55941aca8cf, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:09.156602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecae20a-5a7b-4da0-90a0-841cd0bae169 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:09.176720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1110/2220] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:32:09 2022] PUT /identity/v3/projects/74318a6f074f4d7ea084461473ef1a80/users/0815b4e3389240ada8d1c55941aca8cf/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:09.183266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcda08fb-4c6a-4974-bad9-a5452e83644d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:09.183568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcda08fb-4c6a-4974-bad9-a5452e83644d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:09.183821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcda08fb-4c6a-4974-bad9-a5452e83644d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:09.206694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-fcda08fb-4c6a-4974-bad9-a5452e83644d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:09.208225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-fcda08fb-4c6a-4974-bad9-a5452e83644d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:09.315771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1111/2221] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:32:09.326738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecae20a-5a7b-4da0-90a0-841cd0bae169 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:09.328476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-aecae20a-5a7b-4da0-90a0-841cd0bae169 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:09.430798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecae20a-5a7b-4da0-90a0-841cd0bae169 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:09.435301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa6867-e7c5-4bec-9bd3-d2200fd671bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:09.435979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa6867-e7c5-4bec-9bd3-d2200fd671bf service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:09.436307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa6867-e7c5-4bec-9bd3-d2200fd671bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:09.436589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74aa6867-e7c5-4bec-9bd3-d2200fd671bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:09.455877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aa6867-e7c5-4bec-9bd3-d2200fd671bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:09.459080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74aa6867-e7c5-4bec-9bd3-d2200fd671bf service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:09.478035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1111/2222] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:32:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:32:12.127702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5902d44-433c-485e-80ae-35fcb0b765bc None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:12.127702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5902d44-433c-485e-80ae-35fcb0b765bc None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:12.127702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5902d44-433c-485e-80ae-35fcb0b765bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:12.160820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-f5902d44-433c-485e-80ae-35fcb0b765bc None None] MFA Rules not processed for user `d658f847aadd411390b1d7f7378b8d83`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:12.162089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f5902d44-433c-485e-80ae-35fcb0b765bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:12.241982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1112/2223] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:12 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:12.256234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa6867-e7c5-4bec-9bd3-d2200fd671bf service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:12.257820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-74aa6867-e7c5-4bec-9bd3-d2200fd671bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:12.337698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74aa6867-e7c5-4bec-9bd3-d2200fd671bf service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:12.344702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac02e0e1-ba49-400c-b16d-deb3f4264f1f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:12.345386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac02e0e1-ba49-400c-b16d-deb3f4264f1f service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:12.345685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac02e0e1-ba49-400c-b16d-deb3f4264f1f service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:12.345958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac02e0e1-ba49-400c-b16d-deb3f4264f1f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:12.347948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ac02e0e1-ba49-400c-b16d-deb3f4264f1f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:12.417616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac02e0e1-ba49-400c-b16d-deb3f4264f1f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:12.419258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac02e0e1-ba49-400c-b16d-deb3f4264f1f service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:12.430372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1112/2224] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:32:12 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:32:16.785972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd415b-b5ce-4bc3-8fd7-02e7eeed125e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:16.786543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd415b-b5ce-4bc3-8fd7-02e7eeed125e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:16.786543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd415b-b5ce-4bc3-8fd7-02e7eeed125e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:16.838984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-17cd415b-b5ce-4bc3-8fd7-02e7eeed125e None None] MFA Rules not processed for user `0815b4e3389240ada8d1c55941aca8cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:16.842576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-17cd415b-b5ce-4bc3-8fd7-02e7eeed125e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:16.913443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:16.956990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1113/2225] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:16 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:16.961527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac02e0e1-ba49-400c-b16d-deb3f4264f1f service nova] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:16.995848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac02e0e1-ba49-400c-b16d-deb3f4264f1f service nova] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:17.022337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caa4fce-fe1f-4e45-bd00-b47b85406e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:17.023412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caa4fce-fe1f-4e45-bd00-b47b85406e0a admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:17.024229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caa4fce-fe1f-4e45-bd00-b47b85406e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:17.024720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caa4fce-fe1f-4e45-bd00-b47b85406e0a admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:17.025648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caa4fce-fe1f-4e45-bd00-b47b85406e0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:17.027910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5caa4fce-fe1f-4e45-bd00-b47b85406e0a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:17.082958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1113/2226] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:32:16 2022] POST /identity/v3/projects => generated 385 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:32:17.084138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cd415b-b5ce-4bc3-8fd7-02e7eeed125e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:17.097575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cd415b-b5ce-4bc3-8fd7-02e7eeed125e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:17.182960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd2ff0-018f-46b7-926b-67d12277ef2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:17.185279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd2ff0-018f-46b7-926b-67d12277ef2c admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:17.185678 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd2ff0-018f-46b7-926b-67d12277ef2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:17.186056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd2ff0-018f-46b7-926b-67d12277ef2c admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:17.187147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afd2ff0-018f-46b7-926b-67d12277ef2c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:17.193112 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afd2ff0-018f-46b7-926b-67d12277ef2c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:17.246266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1114/2227] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:32:17 2022] POST /identity/v3/users => generated 299 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:32:17.249917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caa4fce-fe1f-4e45-bd00-b47b85406e0a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:17.263531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caa4fce-fe1f-4e45-bd00-b47b85406e0a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:17.275983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2af4187-4b74-4c1e-b1c0-83421a21aefb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:17.276624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af4187-4b74-4c1e-b1c0-83421a21aefb admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:17.276902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af4187-4b74-4c1e-b1c0-83421a21aefb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:17.277174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af4187-4b74-4c1e-b1c0-83421a21aefb admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:17.277827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2af4187-4b74-4c1e-b1c0-83421a21aefb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:17.284058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2af4187-4b74-4c1e-b1c0-83421a21aefb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:17.287716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:17.339288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1114/2228] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:32:17 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:32:17.339726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd2ff0-018f-46b7-926b-67d12277ef2c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:17.363341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd2ff0-018f-46b7-926b-67d12277ef2c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:17.369586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb4e0eb-e2c9-4761-82ce-a5f9e1ecf8f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:17.370618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb4e0eb-e2c9-4761-82ce-a5f9e1ecf8f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:17.371158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb4e0eb-e2c9-4761-82ce-a5f9e1ecf8f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:17.371827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb4e0eb-e2c9-4761-82ce-a5f9e1ecf8f7 admin admin] PATH_INFO: `/v3/projects/cfaf2e58e0f14b95875b97b38a15dd63/users/e8cbefad6fc24689af293becdfb0f990/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:17.421398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb4e0eb-e2c9-4761-82ce-a5f9e1ecf8f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfaf2e58e0f14b95875b97b38a15dd63, user_id=e8cbefad6fc24689af293becdfb0f990, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:17.424404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb4e0eb-e2c9-4761-82ce-a5f9e1ecf8f7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:17.445462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1115/2229] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:32:17 2022] PUT /identity/v3/projects/cfaf2e58e0f14b95875b97b38a15dd63/users/e8cbefad6fc24689af293becdfb0f990/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:17.449728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2af4187-4b74-4c1e-b1c0-83421a21aefb admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:17.464938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2af4187-4b74-4c1e-b1c0-83421a21aefb admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:17.537965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c102ca4a-8edc-4e3d-9a55-6822e6ade666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:17.538744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c102ca4a-8edc-4e3d-9a55-6822e6ade666 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:17.539436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c102ca4a-8edc-4e3d-9a55-6822e6ade666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:17.539436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c102ca4a-8edc-4e3d-9a55-6822e6ade666 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:17.540952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c102ca4a-8edc-4e3d-9a55-6822e6ade666 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:17.541495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c102ca4a-8edc-4e3d-9a55-6822e6ade666 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:17.554143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1115/2230] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:32:17 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:32:17.558942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb4e0eb-e2c9-4761-82ce-a5f9e1ecf8f7 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:17.573793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb4e0eb-e2c9-4761-82ce-a5f9e1ecf8f7 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:17.579047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc722ef-39f2-4102-8855-d90103efbf34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:17.579676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc722ef-39f2-4102-8855-d90103efbf34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:17.580005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc722ef-39f2-4102-8855-d90103efbf34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:17.580353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc722ef-39f2-4102-8855-d90103efbf34 admin admin] PATH_INFO: `/v3/projects/cfaf2e58e0f14b95875b97b38a15dd63/users/e8cbefad6fc24689af293becdfb0f990/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:17.582394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc722ef-39f2-4102-8855-d90103efbf34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfaf2e58e0f14b95875b97b38a15dd63, user_id=e8cbefad6fc24689af293becdfb0f990, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:17.585431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc722ef-39f2-4102-8855-d90103efbf34 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:17.620995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1116/2231] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:32:17 2022] PUT /identity/v3/projects/cfaf2e58e0f14b95875b97b38a15dd63/users/e8cbefad6fc24689af293becdfb0f990/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:23.907905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4830621-e3d2-483f-80b4-3b72a7e7253f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:23.908465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4830621-e3d2-483f-80b4-3b72a7e7253f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:23.908465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4830621-e3d2-483f-80b4-3b72a7e7253f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:23.948538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d4830621-e3d2-483f-80b4-3b72a7e7253f None None] MFA Rules not processed for user `e8cbefad6fc24689af293becdfb0f990`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:23.950022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d4830621-e3d2-483f-80b4-3b72a7e7253f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:24.048303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1116/2232] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:23 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:24.064026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc722ef-39f2-4102-8855-d90103efbf34 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:24.066182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6dc722ef-39f2-4102-8855-d90103efbf34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:24.138302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc722ef-39f2-4102-8855-d90103efbf34 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:24.142551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d01a89-abfd-47b1-a567-c7197d388bcd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce646dcb8124685ac7a1c1eca664d04', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:24.143183 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d01a89-abfd-47b1-a567-c7197d388bcd service glance] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:24.143456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d01a89-abfd-47b1-a567-c7197d388bcd service glance] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:24.143726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d01a89-abfd-47b1-a567-c7197d388bcd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:24.213103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:24.215712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d01a89-abfd-47b1-a567-c7197d388bcd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:24.217398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d01a89-abfd-47b1-a567-c7197d388bcd service glance] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:24.242562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1117/2233] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:32:24 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:32:24.461802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb39a969-f412-43e3-bd07-a75a87450d14 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:24.462068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb39a969-f412-43e3-bd07-a75a87450d14 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:24.462444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb39a969-f412-43e3-bd07-a75a87450d14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:24.506377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-eb39a969-f412-43e3-bd07-a75a87450d14 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:24.508718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-eb39a969-f412-43e3-bd07-a75a87450d14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:24.594620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1117/2234] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:32:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:24.606524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d01a89-abfd-47b1-a567-c7197d388bcd service glance] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:24.608392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a8d01a89-abfd-47b1-a567-c7197d388bcd service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:24.795181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d01a89-abfd-47b1-a567-c7197d388bcd service glance] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:24.802309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a351ef1-30f8-4667-a1f8-73ee6e21fd23 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:24.803095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a351ef1-30f8-4667-a1f8-73ee6e21fd23 service swift] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:24.803505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a351ef1-30f8-4667-a1f8-73ee6e21fd23 service swift] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:24.803862 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a351ef1-30f8-4667-a1f8-73ee6e21fd23 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:24.821218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a351ef1-30f8-4667-a1f8-73ee6e21fd23 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:24.826022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a351ef1-30f8-4667-a1f8-73ee6e21fd23 service swift] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:24.852242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1118/2235] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:32:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 246 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:32:34.579439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6c6673-754e-4cd3-af17-e5a0cf0f0c20 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:34.580102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6c6673-754e-4cd3-af17-e5a0cf0f0c20 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:34.580102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6c6673-754e-4cd3-af17-e5a0cf0f0c20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:34.584120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=90 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:34.607809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-9f6c6673-754e-4cd3-af17-e5a0cf0f0c20 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:34.609345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9f6c6673-754e-4cd3-af17-e5a0cf0f0c20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:34.617356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1118/2236] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:32:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:34.632243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837172e-e633-42a2-a05f-0bd47b589183 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:34.632243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837172e-e633-42a2-a05f-0bd47b589183 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:34.632243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4837172e-e633-42a2-a05f-0bd47b589183 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:34.645120 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6c6673-754e-4cd3-af17-e5a0cf0f0c20 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:34.660099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6c6673-754e-4cd3-af17-e5a0cf0f0c20 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:34.660303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4837172e-e633-42a2-a05f-0bd47b589183 None None] MFA Rules not processed for user `0815b4e3389240ada8d1c55941aca8cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:34.661517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4837172e-e633-42a2-a05f-0bd47b589183 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:34.665210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629d19b6-3c0c-4434-8c16-a892c0a73746 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:34.667504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629d19b6-3c0c-4434-8c16-a892c0a73746 service swift] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:34.667802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629d19b6-3c0c-4434-8c16-a892c0a73746 service swift] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:34.668073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629d19b6-3c0c-4434-8c16-a892c0a73746 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:34.669181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1119/2237] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:34 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:34.677378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629d19b6-3c0c-4434-8c16-a892c0a73746 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:34.679016 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629d19b6-3c0c-4434-8c16-a892c0a73746 service swift] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:34.687069 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1119/2238] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:32:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 13 15:32:36.717709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bc2cb-8e77-41f9-ac8e-161059475c8f None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:36.718713 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bc2cb-8e77-41f9-ac8e-161059475c8f None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:36.718934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bc2cb-8e77-41f9-ac8e-161059475c8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:36.738913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b46bc2cb-8e77-41f9-ac8e-161059475c8f None None] MFA Rules not processed for user `97189d5c87c94be2bef7cff79c471c7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:36.740879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b46bc2cb-8e77-41f9-ac8e-161059475c8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:36.801531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1120/2239] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:36 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:44.683714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=9 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:46.471892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629d19b6-3c0c-4434-8c16-a892c0a73746 service swift] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:46.473409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-629d19b6-3c0c-4434-8c16-a892c0a73746 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:46.609757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629d19b6-3c0c-4434-8c16-a892c0a73746 service swift] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:46.615883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:46.616965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:46.617590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:46.618073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] PATH_INFO: `/v3/users/97437e4800e249249316bdf988b6e42a` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:46.620541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] RBAC: Authorizing `identity:delete_user(user_id=97437e4800e249249316bdf988b6e42a)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:46.622800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:46.729303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '97437e4800e249249316bdf988b6e42a'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:46.730294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '97437e4800e249249316bdf988b6e42a'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:46.740980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '97437e4800e249249316bdf988b6e42a'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:46.741954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '97437e4800e249249316bdf988b6e42a'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:46.754501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a67f37f3-389f-4cf0-a607-b4d63989dd16 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '97437e4800e249249316bdf988b6e42a'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:46.793792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=62 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:46.805101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1120/2240] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:32:46 2022] DELETE /identity/v3/users/97437e4800e249249316bdf988b6e42a => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:46.812880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c70b3b-e904-4ed4-9d26-f7d8ea8095da None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:46.814635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c70b3b-e904-4ed4-9d26-f7d8ea8095da None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:46.815648 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c70b3b-e904-4ed4-9d26-f7d8ea8095da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:46.884743 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b2c70b3b-e904-4ed4-9d26-f7d8ea8095da None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:46.890918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b2c70b3b-e904-4ed4-9d26-f7d8ea8095da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:46.947623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b75fe50-44c2-4626-b0d8-a746f25fff70 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:46.947870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b75fe50-44c2-4626-b0d8-a746f25fff70 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:46.948111 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b75fe50-44c2-4626-b0d8-a746f25fff70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:46.983782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1121/2241] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:46.984889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3b75fe50-44c2-4626-b0d8-a746f25fff70 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:46.986436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3b75fe50-44c2-4626-b0d8-a746f25fff70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:47.031998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c70b3b-e904-4ed4-9d26-f7d8ea8095da None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:47.033623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b2c70b3b-e904-4ed4-9d26-f7d8ea8095da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:47.093875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1121/2242] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:32:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:32:47.107757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b75fe50-44c2-4626-b0d8-a746f25fff70 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:47.109172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3b75fe50-44c2-4626-b0d8-a746f25fff70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:47.155086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c70b3b-e904-4ed4-9d26-f7d8ea8095da None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:47.159228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a25df6-16b7-4fd6-b0f0-b458085b87f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:47.159947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a25df6-16b7-4fd6-b0f0-b458085b87f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:47.160229 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a25df6-16b7-4fd6-b0f0-b458085b87f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:47.160500 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a25df6-16b7-4fd6-b0f0-b458085b87f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:47.166487 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:47.172165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a25df6-16b7-4fd6-b0f0-b458085b87f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:47.173825 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a25df6-16b7-4fd6-b0f0-b458085b87f8 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:47.187543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1122/2243] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:32:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:32:47.239810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b75fe50-44c2-4626-b0d8-a746f25fff70 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:47.246439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509e6022-d94f-476a-83a4-df06de14bca9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:47.247410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e6022-d94f-476a-83a4-df06de14bca9 service swift] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:47.247410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e6022-d94f-476a-83a4-df06de14bca9 service swift] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:47.247809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e6022-d94f-476a-83a4-df06de14bca9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:47.258465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509e6022-d94f-476a-83a4-df06de14bca9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:47.260187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509e6022-d94f-476a-83a4-df06de14bca9 service swift] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:47.273629 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1122/2244] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:32:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 13 15:32:48.048578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a25df6-16b7-4fd6-b0f0-b458085b87f8 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:48.050727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-17a25df6-16b7-4fd6-b0f0-b458085b87f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:48.098053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a25df6-16b7-4fd6-b0f0-b458085b87f8 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:48.152166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:48.153250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:48.153756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:48.154265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] PATH_INFO: `/v3/projects/73fb66ba3f5942efb8720c416991a6fc` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:48.157637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] RBAC: Authorizing `identity:delete_project(project_id=73fb66ba3f5942efb8720c416991a6fc)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:48.159886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:48.328965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73fb66ba3f5942efb8720c416991a6fc', 'request_id': 'req-eb81811b-ba63-4254-a1c2-fbd50011a947'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:48.336032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73fb66ba3f5942efb8720c416991a6fc', 'request_id': 'req-eb81811b-ba63-4254-a1c2-fbd50011a947'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:48.357630 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73fb66ba3f5942efb8720c416991a6fc', 'request_id': 'req-eb81811b-ba63-4254-a1c2-fbd50011a947'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:48.388079 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] The token cache is being invalidate because project 73fb66ba3f5942efb8720c416991a6fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:32:48.388079 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:48.389413 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eb81811b-ba63-4254-a1c2-fbd50011a947 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:48.397705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1123/2245] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:32:48 2022] DELETE /identity/v3/projects/73fb66ba3f5942efb8720c416991a6fc => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:48.399596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc01bb2-a827-4e76-ae25-20f2fbff01ee None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:48.400185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc01bb2-a827-4e76-ae25-20f2fbff01ee None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:48.400185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc01bb2-a827-4e76-ae25-20f2fbff01ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:48.447904 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-9dc01bb2-a827-4e76-ae25-20f2fbff01ee None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:48.449284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9dc01bb2-a827-4e76-ae25-20f2fbff01ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:48.661411 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1123/2246] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:32:48.669525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d010da15-89c8-4575-9b17-c55d4300fca3 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:48.669843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d010da15-89c8-4575-9b17-c55d4300fca3 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:48.670086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d010da15-89c8-4575-9b17-c55d4300fca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:48.695392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d010da15-89c8-4575-9b17-c55d4300fca3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:48.696759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d010da15-89c8-4575-9b17-c55d4300fca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:48.709795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1124/2247] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:48.711209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc01bb2-a827-4e76-ae25-20f2fbff01ee None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:48.730682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc01bb2-a827-4e76-ae25-20f2fbff01ee None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:48.735973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9280946-bd67-46cb-94ff-297a46837491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:48.736745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9280946-bd67-46cb-94ff-297a46837491 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:48.737172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9280946-bd67-46cb-94ff-297a46837491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:48.737445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9280946-bd67-46cb-94ff-297a46837491 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:48.738277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9280946-bd67-46cb-94ff-297a46837491 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:32:48.738808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9280946-bd67-46cb-94ff-297a46837491 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:48.740473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9280946-bd67-46cb-94ff-297a46837491 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:48.760667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1124/2248] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:32:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:32:48.765857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d010da15-89c8-4575-9b17-c55d4300fca3 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:48.776958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d010da15-89c8-4575-9b17-c55d4300fca3 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:48.787080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d976c1ab-049f-434a-a419-55340e360d85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:48.787635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d976c1ab-049f-434a-a419-55340e360d85 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:48.787915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d976c1ab-049f-434a-a419-55340e360d85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:48.788188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d976c1ab-049f-434a-a419-55340e360d85 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:48.788868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d976c1ab-049f-434a-a419-55340e360d85 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:48.811433 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d976c1ab-049f-434a-a419-55340e360d85 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:48.836624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:48.845160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1125/2249] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:32:48 2022] POST /identity/v3/projects => generated 383 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:32:48.849720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9280946-bd67-46cb-94ff-297a46837491 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:48.862214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9280946-bd67-46cb-94ff-297a46837491 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:48.958142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85f764d-9db8-48e6-a42e-a780a9cc86a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:48.959254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85f764d-9db8-48e6-a42e-a780a9cc86a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:48.959695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85f764d-9db8-48e6-a42e-a780a9cc86a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:48.960024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85f764d-9db8-48e6-a42e-a780a9cc86a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:48.960679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85f764d-9db8-48e6-a42e-a780a9cc86a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:48.962773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85f764d-9db8-48e6-a42e-a780a9cc86a2 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:48.992596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1125/2250] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:32:48 2022] POST /identity/v3/users => generated 292 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:32:48.998888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d976c1ab-049f-434a-a419-55340e360d85 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:49.015273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d976c1ab-049f-434a-a419-55340e360d85 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:49.021204 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d137c6c-d41c-48a6-8fa8-15f463037b7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:49.022049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d137c6c-d41c-48a6-8fa8-15f463037b7c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:49.022399 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d137c6c-d41c-48a6-8fa8-15f463037b7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:49.022738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d137c6c-d41c-48a6-8fa8-15f463037b7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:49.023498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d137c6c-d41c-48a6-8fa8-15f463037b7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:49.025139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d137c6c-d41c-48a6-8fa8-15f463037b7c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:49.041693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1126/2251] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:32:48 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:32:49.043229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85f764d-9db8-48e6-a42e-a780a9cc86a2 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:49.074280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85f764d-9db8-48e6-a42e-a780a9cc86a2 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:49.080457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af48bfca-ab16-4cab-be0a-ead2de328511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:49.081088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af48bfca-ab16-4cab-be0a-ead2de328511 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:49.081456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af48bfca-ab16-4cab-be0a-ead2de328511 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:49.081639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af48bfca-ab16-4cab-be0a-ead2de328511 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:49.082475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af48bfca-ab16-4cab-be0a-ead2de328511 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:49.084134 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af48bfca-ab16-4cab-be0a-ead2de328511 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:49.098151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1126/2252] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:32:49 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:32:49.101835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d137c6c-d41c-48a6-8fa8-15f463037b7c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:49.112768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d137c6c-d41c-48a6-8fa8-15f463037b7c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:49.118222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6721679d-fa43-443b-8a12-e7b0ebe4c839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:49.118938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6721679d-fa43-443b-8a12-e7b0ebe4c839 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:49.119214 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6721679d-fa43-443b-8a12-e7b0ebe4c839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:49.122315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6721679d-fa43-443b-8a12-e7b0ebe4c839 admin admin] PATH_INFO: `/v3/projects/ffb44234e9014eff98a61cda9facc146/users/78228bf8cfb34a9fbcf954a42638a339/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:49.139505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6721679d-fa43-443b-8a12-e7b0ebe4c839 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffb44234e9014eff98a61cda9facc146, user_id=78228bf8cfb34a9fbcf954a42638a339, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:49.149769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6721679d-fa43-443b-8a12-e7b0ebe4c839 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:49.168011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1127/2253] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:32:49 2022] PUT /identity/v3/projects/ffb44234e9014eff98a61cda9facc146/users/78228bf8cfb34a9fbcf954a42638a339/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:49.180123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2388183d-f575-420d-b537-f3f6ec693a44 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:49.180789 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2388183d-f575-420d-b537-f3f6ec693a44 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:49.181164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2388183d-f575-420d-b537-f3f6ec693a44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:49.221912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2388183d-f575-420d-b537-f3f6ec693a44 None None] MFA Rules not processed for user `78228bf8cfb34a9fbcf954a42638a339`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:49.223857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2388183d-f575-420d-b537-f3f6ec693a44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:49.233202 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:49.316304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1127/2254] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:49.329198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6721679d-fa43-443b-8a12-e7b0ebe4c839 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:49.331231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6721679d-fa43-443b-8a12-e7b0ebe4c839 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:49.407226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6721679d-fa43-443b-8a12-e7b0ebe4c839 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:49.415987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce67dac2-f217-4297-88c4-d16db38091d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:49.416633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67dac2-f217-4297-88c4-d16db38091d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:49.416909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67dac2-f217-4297-88c4-d16db38091d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:49.417185 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67dac2-f217-4297-88c4-d16db38091d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:49.426642 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce67dac2-f217-4297-88c4-d16db38091d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:49.428077 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce67dac2-f217-4297-88c4-d16db38091d1 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:49.451046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1128/2255] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:32:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:32:53.348628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2388183d-f575-420d-b537-f3f6ec693a44 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:53.452820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2388183d-f575-420d-b537-f3f6ec693a44 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:53.457656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:53.458507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:53.458772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:53.459113 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] PATH_INFO: `/v3/users/78228bf8cfb34a9fbcf954a42638a339` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:53.461255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78228bf8cfb34a9fbcf954a42638a339)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:53.472337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:53.541439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78228bf8cfb34a9fbcf954a42638a339'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:53.542179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78228bf8cfb34a9fbcf954a42638a339'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:53.550022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78228bf8cfb34a9fbcf954a42638a339'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:53.550639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78228bf8cfb34a9fbcf954a42638a339'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:53.562913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78228bf8cfb34a9fbcf954a42638a339'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:53.665075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1128/2256] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:32:53 2022] DELETE /identity/v3/users/78228bf8cfb34a9fbcf954a42638a339 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:53.669021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a633719-ef94-401e-9479-0fecb3631841 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:53.669649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a633719-ef94-401e-9479-0fecb3631841 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:53.669649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a633719-ef94-401e-9479-0fecb3631841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:53.774158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8a633719-ef94-401e-9479-0fecb3631841 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:53.774158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8a633719-ef94-401e-9479-0fecb3631841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:53.909369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1129/2257] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:32:53.919758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:53.921235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:53.999230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:54.057630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587a7bb0-02ae-4902-9b39-ef100ae77ab3 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:54.064055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86de526f-6387-4828-b238-4b9dfc9d959e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:54.064787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86de526f-6387-4828-b238-4b9dfc9d959e service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:54.065073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86de526f-6387-4828-b238-4b9dfc9d959e service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:54.065350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86de526f-6387-4828-b238-4b9dfc9d959e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:54.095585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86de526f-6387-4828-b238-4b9dfc9d959e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:54.097253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86de526f-6387-4828-b238-4b9dfc9d959e service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:54.131678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1129/2258] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:32:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:32:54.649652 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a633719-ef94-401e-9479-0fecb3631841 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:32:54.651078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8a633719-ef94-401e-9479-0fecb3631841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:54.685950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a633719-ef94-401e-9479-0fecb3631841 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:32:54.691055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:32:54.691994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:54.692574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:54.693090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] PATH_INFO: `/v3/projects/ffb44234e9014eff98a61cda9facc146` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:54.694590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffb44234e9014eff98a61cda9facc146)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:32:54.705618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:32:54.788497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffb44234e9014eff98a61cda9facc146', 'request_id': 'req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:54.798012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffb44234e9014eff98a61cda9facc146', 'request_id': 'req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:54.807758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:32:54.858868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffb44234e9014eff98a61cda9facc146', 'request_id': 'req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:54.876192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] The token cache is being invalidate because project ffb44234e9014eff98a61cda9facc146 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:32:54.876518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:54.877147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-9a34ff2a-e931-4ee6-845e-4de82b5e5dd4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:32:54.878714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1130/2259] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:32:54 2022] DELETE /identity/v3/projects/ffb44234e9014eff98a61cda9facc146 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:32:56.296746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21af5bca-fec8-40f6-8eb3-656c4ccb1664 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:32:56.297369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21af5bca-fec8-40f6-8eb3-656c4ccb1664 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:32:56.297369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21af5bca-fec8-40f6-8eb3-656c4ccb1664 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:32:56.335962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-21af5bca-fec8-40f6-8eb3-656c4ccb1664 None None] MFA Rules not processed for user `1dbbc500cd8c4cedb3784d7750cb6de2`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:32:56.337359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-21af5bca-fec8-40f6-8eb3-656c4ccb1664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:32:56.414171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1130/2260] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:32:56 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:33:00.214369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614e1562-ad42-4b03-b5a6-d18b99225667 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:33:00.214992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614e1562-ad42-4b03-b5a6-d18b99225667 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:33:00.215101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614e1562-ad42-4b03-b5a6-d18b99225667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:33:00.236758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-614e1562-ad42-4b03-b5a6-d18b99225667 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:33:00.239024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-614e1562-ad42-4b03-b5a6-d18b99225667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:33:00.311162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1131/2261] 10.222.0.16 () {58 vars in 967 bytes} [Tue Sep 13 15:33:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:33:00.324066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21af5bca-fec8-40f6-8eb3-656c4ccb1664 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:33:00.325495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-21af5bca-fec8-40f6-8eb3-656c4ccb1664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:33:00.404505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21af5bca-fec8-40f6-8eb3-656c4ccb1664 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:33:00.411606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23866cb1-ce4b-438e-87b7-13bc0d6e306b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:33:00.412676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23866cb1-ce4b-438e-87b7-13bc0d6e306b service swift] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:33:00.413026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23866cb1-ce4b-438e-87b7-13bc0d6e306b service swift] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:33:00.414232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23866cb1-ce4b-438e-87b7-13bc0d6e306b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:33:00.424261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23866cb1-ce4b-438e-87b7-13bc0d6e306b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:33:00.426224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23866cb1-ce4b-438e-87b7-13bc0d6e306b service swift] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:33:00.435553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1131/2262] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:33:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:33:02.772069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614e1562-ad42-4b03-b5a6-d18b99225667 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:33:02.773668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-614e1562-ad42-4b03-b5a6-d18b99225667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:33:02.847152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614e1562-ad42-4b03-b5a6-d18b99225667 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:33:02.851924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:33:02.852564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:33:02.852840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:33:02.853148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] PATH_INFO: `/v3/users/5ae72d51e9004670a50362d1206fab21` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:33:02.854357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ae72d51e9004670a50362d1206fab21)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:33:02.856318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:33:02.911942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ae72d51e9004670a50362d1206fab21'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:33:02.920728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ae72d51e9004670a50362d1206fab21'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:33:02.921450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ae72d51e9004670a50362d1206fab21'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:33:02.922199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ae72d51e9004670a50362d1206fab21'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:33:02.929787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ae72d51e9004670a50362d1206fab21'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:33:02.948883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1132/2263] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:33:02 2022] DELETE /identity/v3/users/5ae72d51e9004670a50362d1206fab21 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:33:03.128243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23866cb1-ce4b-438e-87b7-13bc0d6e306b service swift] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:33:03.139585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23866cb1-ce4b-438e-87b7-13bc0d6e306b service swift] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:33:03.197396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:33:03.197876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:33:03.198178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:33:03.198448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] PATH_INFO: `/v3/projects/5b820ea465db4802aaa22dbaa4e7397a` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:33:03.199617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b820ea465db4802aaa22dbaa4e7397a)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:33:03.201537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:33:03.210195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:33:03.251644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b820ea465db4802aaa22dbaa4e7397a', 'request_id': 'req-521c64a5-8c09-45fd-b858-b9f7db962874'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:33:03.257233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b820ea465db4802aaa22dbaa4e7397a', 'request_id': 'req-521c64a5-8c09-45fd-b858-b9f7db962874'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:33:03.267589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b820ea465db4802aaa22dbaa4e7397a', 'request_id': 'req-521c64a5-8c09-45fd-b858-b9f7db962874'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:33:03.279199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] The token cache is being invalidate because project 5b820ea465db4802aaa22dbaa4e7397a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:33:03.279478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:33:03.280089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:33:03.281553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1132/2264] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:33:03 2022] DELETE /identity/v3/projects/5b820ea465db4802aaa22dbaa4e7397a => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:33:11.859633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:33:11.861655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:33:11.881615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:33:11.949154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f634f49d-9eff-4748-acb4-4f4105e6c022 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:33:12.015271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f9ef0f-a462-4051-b8e7-7521abe8d423 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:33:12.016309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f9ef0f-a462-4051-b8e7-7521abe8d423 service placement] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:33:12.016785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f9ef0f-a462-4051-b8e7-7521abe8d423 service placement] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:33:12.017266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f9ef0f-a462-4051-b8e7-7521abe8d423 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:33:12.020253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-55f9ef0f-a462-4051-b8e7-7521abe8d423 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:33:12.039138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f9ef0f-a462-4051-b8e7-7521abe8d423 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:33:12.040855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f9ef0f-a462-4051-b8e7-7521abe8d423 service placement] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:33:12.053061 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1133/2265] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:33:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:33:13.276047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=21,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:33:22.048710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=52 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:33:27.209740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:33:27.211357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:33:27.273041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521c64a5-8c09-45fd-b858-b9f7db962874 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:33:27.276713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ee799d-cdfa-4b24-ab57-3aa308626e6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:33:27.277326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ee799d-cdfa-4b24-ab57-3aa308626e6f service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:33:27.277598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ee799d-cdfa-4b24-ab57-3aa308626e6f service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:33:27.277867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ee799d-cdfa-4b24-ab57-3aa308626e6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:33:27.279667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-23ee799d-cdfa-4b24-ab57-3aa308626e6f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:33:27.347492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ee799d-cdfa-4b24-ab57-3aa308626e6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:33:27.348921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ee799d-cdfa-4b24-ab57-3aa308626e6f service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:33:27.358828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1133/2266] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:33:27 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:33:37.354341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=50 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:33:51.115499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd0f5d-d3b1-428f-a5f7-69063330c8c5 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:33:51.115998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd0f5d-d3b1-428f-a5f7-69063330c8c5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:33:51.115998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fd0f5d-d3b1-428f-a5f7-69063330c8c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:33:51.135992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-96fd0f5d-d3b1-428f-a5f7-69063330c8c5 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:33:51.137205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-96fd0f5d-d3b1-428f-a5f7-69063330c8c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:33:51.196338 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1134/2267] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:33:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:33:51.219131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ee799d-cdfa-4b24-ab57-3aa308626e6f service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:33:51.220770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-23ee799d-cdfa-4b24-ab57-3aa308626e6f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:33:51.291795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ee799d-cdfa-4b24-ab57-3aa308626e6f service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:33:51.296995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df2e8fb-1549-4ef9-8cc4-fc66c4898294 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:33:51.297607 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df2e8fb-1549-4ef9-8cc4-fc66c4898294 service swift] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:33:51.297882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df2e8fb-1549-4ef9-8cc4-fc66c4898294 service swift] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:33:51.298174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df2e8fb-1549-4ef9-8cc4-fc66c4898294 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:33:51.306854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df2e8fb-1549-4ef9-8cc4-fc66c4898294 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:33:51.308509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df2e8fb-1549-4ef9-8cc4-fc66c4898294 service swift] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:33:51.317316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1134/2268] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:33:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:34:01.188974 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=25 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:34:01.313948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=32 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:34:05.895857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fd0f5d-d3b1-428f-a5f7-69063330c8c5 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:34:05.897309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-96fd0f5d-d3b1-428f-a5f7-69063330c8c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:05.967168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fd0f5d-d3b1-428f-a5f7-69063330c8c5 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:34:05.972805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:34:05.973921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:05.974335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:05.974649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] PATH_INFO: `/v3/users/1dbbc500cd8c4cedb3784d7750cb6de2` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:05.976519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] RBAC: Authorizing `identity:delete_user(user_id=1dbbc500cd8c4cedb3784d7750cb6de2)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:34:05.979174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:34:06.037447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1dbbc500cd8c4cedb3784d7750cb6de2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.045601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1dbbc500cd8c4cedb3784d7750cb6de2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.046280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1dbbc500cd8c4cedb3784d7750cb6de2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.046956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1dbbc500cd8c4cedb3784d7750cb6de2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.061509 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1dbbc500cd8c4cedb3784d7750cb6de2'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.080363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1135/2269] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:34:05 2022] DELETE /identity/v3/users/1dbbc500cd8c4cedb3784d7750cb6de2 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:34:06.267054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df2e8fb-1549-4ef9-8cc4-fc66c4898294 service swift] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:34:06.282989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df2e8fb-1549-4ef9-8cc4-fc66c4898294 service swift] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:34:06.350895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:34:06.351490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:06.351770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:06.352046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] PATH_INFO: `/v3/projects/f99e1e9c3c13491e86734af2b526fc42` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:06.353164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] RBAC: Authorizing `identity:delete_project(project_id=f99e1e9c3c13491e86734af2b526fc42)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:34:06.355263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:34:06.403137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f99e1e9c3c13491e86734af2b526fc42', 'request_id': 'req-d3478192-6fef-4607-ac89-1a9f09b6ee9e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.408774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f99e1e9c3c13491e86734af2b526fc42', 'request_id': 'req-d3478192-6fef-4607-ac89-1a9f09b6ee9e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.417520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f99e1e9c3c13491e86734af2b526fc42', 'request_id': 'req-d3478192-6fef-4607-ac89-1a9f09b6ee9e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.430212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] The token cache is being invalidate because project f99e1e9c3c13491e86734af2b526fc42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:34:06.430498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.431161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.432642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1135/2270] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:34:06 2022] DELETE /identity/v3/projects/f99e1e9c3c13491e86734af2b526fc42 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:34:06.436829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:34:06.438618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:06.451925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fc5c3d-4346-40d3-b067-ba798665fa8c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:34:06.513716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:34:06.514424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:06.514688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:06.514962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] PATH_INFO: `/v3/users/d658f847aadd411390b1d7f7378b8d83` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:06.516211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=d658f847aadd411390b1d7f7378b8d83)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:34:06.518551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:34:06.540495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:34:06.572723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd658f847aadd411390b1d7f7378b8d83'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.581828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd658f847aadd411390b1d7f7378b8d83'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.582672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd658f847aadd411390b1d7f7378b8d83'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.583232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd658f847aadd411390b1d7f7378b8d83'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.591485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3a4435d6-69af-41af-a866-27bbea1345bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd658f847aadd411390b1d7f7378b8d83'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.609139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1136/2271] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:34:06 2022] DELETE /identity/v3/users/d658f847aadd411390b1d7f7378b8d83 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:34:06.801189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:34:06.803083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:06.848948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3478192-6fef-4607-ac89-1a9f09b6ee9e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:34:06.915968 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:34:06.918190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:06.918190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:06.918190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] PATH_INFO: `/v3/projects/dc9ec7f984554ee087775cb0d36aed7c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:06.918621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc9ec7f984554ee087775cb0d36aed7c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:34:06.920745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:34:06.969983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:34:06.985511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc9ec7f984554ee087775cb0d36aed7c', 'request_id': 'req-be16b98c-de12-4c01-912f-3ac71ea2646c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:06.998653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc9ec7f984554ee087775cb0d36aed7c', 'request_id': 'req-be16b98c-de12-4c01-912f-3ac71ea2646c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:07.010862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc9ec7f984554ee087775cb0d36aed7c', 'request_id': 'req-be16b98c-de12-4c01-912f-3ac71ea2646c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:07.026542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] The token cache is being invalidate because project dc9ec7f984554ee087775cb0d36aed7c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:34:07.026838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:07.027542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-be16b98c-de12-4c01-912f-3ac71ea2646c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:34:07.042771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1136/2272] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:34:06 2022] DELETE /identity/v3/projects/dc9ec7f984554ee087775cb0d36aed7c => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:34:07.281272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b351e-df4e-4419-a64f-64e62717d919 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:07.282306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b351e-df4e-4419-a64f-64e62717d919 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:07.282828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b351e-df4e-4419-a64f-64e62717d919 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:07.285039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1137/2273] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:34:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:34:07.290760 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ad966d-2eb6-42d5-9c64-d5c5fc929767 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:07.291290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ad966d-2eb6-42d5-9c64-d5c5fc929767 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:07.291290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ad966d-2eb6-42d5-9c64-d5c5fc929767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:07.317750 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-45ad966d-2eb6-42d5-9c64-d5c5fc929767 None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:34:07.318772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-45ad966d-2eb6-42d5-9c64-d5c5fc929767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:07.443847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1137/2274] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:34:07 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:34:07.459737 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b351e-df4e-4419-a64f-64e62717d919 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:34:07.461905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-da2b351e-df4e-4419-a64f-64e62717d919 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:07.541038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b351e-df4e-4419-a64f-64e62717d919 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:34:07.544653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48444605-fdba-4810-80df-328eaf388ff3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce646dcb8124685ac7a1c1eca664d04', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:34:07.545268 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48444605-fdba-4810-80df-328eaf388ff3 service glance] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:07.545538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48444605-fdba-4810-80df-328eaf388ff3 service glance] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:07.545799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48444605-fdba-4810-80df-328eaf388ff3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:07.557491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48444605-fdba-4810-80df-328eaf388ff3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:34:07.559345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48444605-fdba-4810-80df-328eaf388ff3 service glance] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:34:07.570520 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1138/2275] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:34:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:34:07.657576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4307e239-3333-4de1-8c9b-68ee5ec20f36 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:07.657853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4307e239-3333-4de1-8c9b-68ee5ec20f36 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:07.658140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4307e239-3333-4de1-8c9b-68ee5ec20f36 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:07.659568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1138/2276] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:34:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 13 15:34:07.664404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb5194b-0c04-40cf-a559-d00c15fa1526 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:07.664689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb5194b-0c04-40cf-a559-d00c15fa1526 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:07.664956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb5194b-0c04-40cf-a559-d00c15fa1526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:07.687612 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4eb5194b-0c04-40cf-a559-d00c15fa1526 None None] MFA Rules not processed for user `171e916244804add8f18b44b77b03d72`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:34:07.688981 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4eb5194b-0c04-40cf-a559-d00c15fa1526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:07.743317 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1139/2277] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:34:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:34:07.755579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4307e239-3333-4de1-8c9b-68ee5ec20f36 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:34:07.757523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4307e239-3333-4de1-8c9b-68ee5ec20f36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:07.772136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4307e239-3333-4de1-8c9b-68ee5ec20f36 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:34:07.775788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaae02c7-d765-437d-a130-c58c634a8845 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:34:07.776384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae02c7-d765-437d-a130-c58c634a8845 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:07.776660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae02c7-d765-437d-a130-c58c634a8845 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:07.776925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae02c7-d765-437d-a130-c58c634a8845 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:07.784395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaae02c7-d765-437d-a130-c58c634a8845 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:34:07.785850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaae02c7-d765-437d-a130-c58c634a8845 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:34:07.793981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1139/2278] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:34:07 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:34:07.884979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39015cb-c4b0-4995-96de-b8db36b91cb0 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:07.885274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39015cb-c4b0-4995-96de-b8db36b91cb0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:07.885544 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39015cb-c4b0-4995-96de-b8db36b91cb0 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:07.887014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1140/2279] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:34:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:34:07.892264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87b17aa-69be-4a79-bff6-662095f7dc84 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:07.892547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87b17aa-69be-4a79-bff6-662095f7dc84 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:07.892814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87b17aa-69be-4a79-bff6-662095f7dc84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:07.914643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-a87b17aa-69be-4a79-bff6-662095f7dc84 None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:34:07.915782 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a87b17aa-69be-4a79-bff6-662095f7dc84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:07.925995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1140/2280] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:34:07 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:34:07.937415 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39015cb-c4b0-4995-96de-b8db36b91cb0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:34:07.939252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c39015cb-c4b0-4995-96de-b8db36b91cb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:07.953170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39015cb-c4b0-4995-96de-b8db36b91cb0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:34:07.956773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863a38b1-4854-4d9c-b64f-fa7c88161fa9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:34:07.957389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863a38b1-4854-4d9c-b64f-fa7c88161fa9 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:07.957663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863a38b1-4854-4d9c-b64f-fa7c88161fa9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:07.957931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863a38b1-4854-4d9c-b64f-fa7c88161fa9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:07.970007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863a38b1-4854-4d9c-b64f-fa7c88161fa9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:34:07.972081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863a38b1-4854-4d9c-b64f-fa7c88161fa9 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:34:07.983187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1141/2281] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:34:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:34:10.706271 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdeff1e-a672-4c98-9c22-640e3c30130e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:10.706817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdeff1e-a672-4c98-9c22-640e3c30130e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:10.706817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdeff1e-a672-4c98-9c22-640e3c30130e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:10.727409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-bbdeff1e-a672-4c98-9c22-640e3c30130e None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:34:10.728624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-bbdeff1e-a672-4c98-9c22-640e3c30130e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:10.786575 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1141/2282] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:34:10 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:34:10.799211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863a38b1-4854-4d9c-b64f-fa7c88161fa9 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:34:10.800804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-863a38b1-4854-4d9c-b64f-fa7c88161fa9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:10.846308 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:34:10.878465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863a38b1-4854-4d9c-b64f-fa7c88161fa9 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:34:10.883688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a8d0bd-0740-4058-93a5-124cb0de0cd9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:34:10.884299 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8d0bd-0740-4058-93a5-124cb0de0cd9 service swift] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:10.884575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8d0bd-0740-4058-93a5-124cb0de0cd9 service swift] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:10.884847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8d0bd-0740-4058-93a5-124cb0de0cd9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:10.893442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a8d0bd-0740-4058-93a5-124cb0de0cd9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:34:10.895154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a8d0bd-0740-4058-93a5-124cb0de0cd9 service swift] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:34:10.903448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1142/2283] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:34:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:34:20.778624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=77,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:34:20.900172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=15 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:34:37.321834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770c83ed-5655-49f0-9579-7db898b4ac95 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:37.322425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770c83ed-5655-49f0-9579-7db898b4ac95 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:37.322692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770c83ed-5655-49f0-9579-7db898b4ac95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:37.342762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-770c83ed-5655-49f0-9579-7db898b4ac95 None None] MFA Rules not processed for user `f48ce3ac32e741ef98e5025fd23bbaa0`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:34:37.344067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-770c83ed-5655-49f0-9579-7db898b4ac95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:37.400870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1142/2284] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:34:37 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:34:40.347569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e412e9-7145-4e39-8acf-d95a2ec96a3c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:40.347569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e412e9-7145-4e39-8acf-d95a2ec96a3c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:40.348376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e412e9-7145-4e39-8acf-d95a2ec96a3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:40.387836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e3e412e9-7145-4e39-8acf-d95a2ec96a3c None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:34:40.388668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e3e412e9-7145-4e39-8acf-d95a2ec96a3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:40.396496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1143/2285] 10.222.0.16 () {58 vars in 967 bytes} [Tue Sep 13 15:34:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:34:40.410002 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770c83ed-5655-49f0-9579-7db898b4ac95 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:34:40.424040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770c83ed-5655-49f0-9579-7db898b4ac95 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:34:40.429659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e9706e-a251-4d52-aa2f-752d727459c7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:34:40.430338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e9706e-a251-4d52-aa2f-752d727459c7 service swift] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:40.430612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e9706e-a251-4d52-aa2f-752d727459c7 service swift] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:40.430872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e9706e-a251-4d52-aa2f-752d727459c7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:40.440689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e9706e-a251-4d52-aa2f-752d727459c7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:34:40.442321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e9706e-a251-4d52-aa2f-752d727459c7 service swift] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:34:40.451985 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1143/2286] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:34:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:34:50.381848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:34:50.447413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=31 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:34:57.321556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e412e9-7145-4e39-8acf-d95a2ec96a3c None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:34:57.331529 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e412e9-7145-4e39-8acf-d95a2ec96a3c None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:34:57.335954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a176d26-4c7e-4a21-ac09-3c03c7381d45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:34:57.336561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a176d26-4c7e-4a21-ac09-3c03c7381d45 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:34:57.337224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a176d26-4c7e-4a21-ac09-3c03c7381d45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:34:57.337515 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a176d26-4c7e-4a21-ac09-3c03c7381d45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:34:57.339465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6a176d26-4c7e-4a21-ac09-3c03c7381d45 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:34:57.353307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a176d26-4c7e-4a21-ac09-3c03c7381d45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:34:57.354833 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a176d26-4c7e-4a21-ac09-3c03c7381d45 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:34:57.365788 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1144/2287] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:34:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:35:07.360956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=6 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:35:08.310309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e9706e-a251-4d52-aa2f-752d727459c7 service swift] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:35:08.311936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e6e9706e-a251-4d52-aa2f-752d727459c7 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:35:08.328347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e9706e-a251-4d52-aa2f-752d727459c7 service swift] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:35:08.335227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bf3f45-d9f6-4f3f-af31-e8b76a4fa579 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:35:08.335821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bf3f45-d9f6-4f3f-af31-e8b76a4fa579 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:35:08.336101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bf3f45-d9f6-4f3f-af31-e8b76a4fa579 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:35:08.336709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bf3f45-d9f6-4f3f-af31-e8b76a4fa579 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:35:08.338582 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b3bf3f45-d9f6-4f3f-af31-e8b76a4fa579 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:35:08.533742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bf3f45-d9f6-4f3f-af31-e8b76a4fa579 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:35:08.535269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bf3f45-d9f6-4f3f-af31-e8b76a4fa579 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:35:08.544486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1144/2288] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:35:08 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:35:13.974599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a176d26-4c7e-4a21-ac09-3c03c7381d45 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:35:13.986615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a176d26-4c7e-4a21-ac09-3c03c7381d45 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:35:13.992693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3871e413-8772-4901-822d-558307194566 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:35:13.993307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871e413-8772-4901-822d-558307194566 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:35:13.993575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871e413-8772-4901-822d-558307194566 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:35:13.993860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871e413-8772-4901-822d-558307194566 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:35:13.995609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3871e413-8772-4901-822d-558307194566 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:35:14.065650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3871e413-8772-4901-822d-558307194566 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:35:14.067265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3871e413-8772-4901-822d-558307194566 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:35:14.076695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1145/2289] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:35:13 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:35:18.540397 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=28 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:35:24.072626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=28 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:35:25.248357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bf3f45-d9f6-4f3f-af31-e8b76a4fa579 service nova] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:35:25.258933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bf3f45-d9f6-4f3f-af31-e8b76a4fa579 service nova] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:35:25.263709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9815b52-4c81-4701-be81-6a07ad50a095 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:35:25.264319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9815b52-4c81-4701-be81-6a07ad50a095 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:35:25.264583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9815b52-4c81-4701-be81-6a07ad50a095 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:35:25.264843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9815b52-4c81-4701-be81-6a07ad50a095 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:35:25.267071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f9815b52-4c81-4701-be81-6a07ad50a095 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:35:25.278027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9815b52-4c81-4701-be81-6a07ad50a095 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:35:25.279494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9815b52-4c81-4701-be81-6a07ad50a095 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:35:25.288231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1145/2290] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:35:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:35:27.373871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560cc0f1-ba79-456a-8447-5ff376242c95 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:35:27.374466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560cc0f1-ba79-456a-8447-5ff376242c95 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:35:27.374466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560cc0f1-ba79-456a-8447-5ff376242c95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:35:27.401475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-560cc0f1-ba79-456a-8447-5ff376242c95 None None] MFA Rules not processed for user `97189d5c87c94be2bef7cff79c471c7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:35:27.402810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-560cc0f1-ba79-456a-8447-5ff376242c95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:35:27.410631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1146/2291] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:35:27 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:35:33.796511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9815b52-4c81-4701-be81-6a07ad50a095 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:35:33.811371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9815b52-4c81-4701-be81-6a07ad50a095 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:35:33.817542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c817a0-92b4-49ec-8134-a8469a7558f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:35:33.818206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c817a0-92b4-49ec-8134-a8469a7558f4 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:35:33.818746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c817a0-92b4-49ec-8134-a8469a7558f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:35:33.819012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c817a0-92b4-49ec-8134-a8469a7558f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:35:33.821003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c4c817a0-92b4-49ec-8134-a8469a7558f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:35:33.835477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c817a0-92b4-49ec-8134-a8469a7558f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:35:33.837103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c817a0-92b4-49ec-8134-a8469a7558f4 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:35:33.850558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1146/2292] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:35:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:35:37.395410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:35:43.844068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=12 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:35:58.679234 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560cc0f1-ba79-456a-8447-5ff376242c95 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:35:58.693506 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560cc0f1-ba79-456a-8447-5ff376242c95 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:35:58.699179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da49f588-c89b-4429-978b-393b9462f049 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:35:58.699776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49f588-c89b-4429-978b-393b9462f049 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:35:58.700066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49f588-c89b-4429-978b-393b9462f049 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:35:58.700313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49f588-c89b-4429-978b-393b9462f049 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:35:58.702230 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-da49f588-c89b-4429-978b-393b9462f049 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:35:58.762778 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da49f588-c89b-4429-978b-393b9462f049 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:35:58.764590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da49f588-c89b-4429-978b-393b9462f049 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:35:58.775577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1147/2293] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:35:58 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:36:03.187197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c817a0-92b4-49ec-8134-a8469a7558f4 service nova] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:36:03.199882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c817a0-92b4-49ec-8134-a8469a7558f4 service nova] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:36:03.207319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045ecf11-7720-446c-9549-759e2858e113 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:36:03.207946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045ecf11-7720-446c-9549-759e2858e113 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:03.208280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045ecf11-7720-446c-9549-759e2858e113 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:03.208552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045ecf11-7720-446c-9549-759e2858e113 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:03.211119 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-045ecf11-7720-446c-9549-759e2858e113 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:03.224983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045ecf11-7720-446c-9549-759e2858e113 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:36:03.226635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045ecf11-7720-446c-9549-759e2858e113 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:36:03.239570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1147/2294] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:36:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:36:06.043053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4ed460-a0e1-418a-9abb-4bc4d32f82c9 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:06.043585 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4ed460-a0e1-418a-9abb-4bc4d32f82c9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:06.043585 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4ed460-a0e1-418a-9abb-4bc4d32f82c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:06.067872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-fa4ed460-a0e1-418a-9abb-4bc4d32f82c9 None None] MFA Rules not processed for user `515e0f8a3a234deba5f909271b303abb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:36:06.069082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-fa4ed460-a0e1-418a-9abb-4bc4d32f82c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:06.077174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1148/2295] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:36:06 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:36:06.410797 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045ecf11-7720-446c-9549-759e2858e113 service nova] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:36:06.422473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045ecf11-7720-446c-9549-759e2858e113 service nova] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:36:06.427103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22cf664-371c-44b7-bb8e-0030aff5d9c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:36:06.427774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22cf664-371c-44b7-bb8e-0030aff5d9c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:06.428044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22cf664-371c-44b7-bb8e-0030aff5d9c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:06.428350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22cf664-371c-44b7-bb8e-0030aff5d9c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:06.430431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e22cf664-371c-44b7-bb8e-0030aff5d9c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:06.441697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22cf664-371c-44b7-bb8e-0030aff5d9c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:36:06.443143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22cf664-371c-44b7-bb8e-0030aff5d9c4 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:36:06.450967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1148/2296] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:36:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:36:06.782080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4ed460-a0e1-418a-9abb-4bc4d32f82c9 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:36:06.783917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-fa4ed460-a0e1-418a-9abb-4bc4d32f82c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:06.857843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4ed460-a0e1-418a-9abb-4bc4d32f82c9 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:36:06.863345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e364f1cc-1b0c-4e84-aed5-2ae7a4982ef4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:36:06.863956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e364f1cc-1b0c-4e84-aed5-2ae7a4982ef4 service placement] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:06.864231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e364f1cc-1b0c-4e84-aed5-2ae7a4982ef4 service placement] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:06.864495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e364f1cc-1b0c-4e84-aed5-2ae7a4982ef4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:06.866358 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e364f1cc-1b0c-4e84-aed5-2ae7a4982ef4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:06.879228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e364f1cc-1b0c-4e84-aed5-2ae7a4982ef4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:36:06.881752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e364f1cc-1b0c-4e84-aed5-2ae7a4982ef4 service placement] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:36:06.898390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1149/2297] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:36:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:36:16.144198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22cf664-371c-44b7-bb8e-0030aff5d9c4 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:36:16.159513 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22cf664-371c-44b7-bb8e-0030aff5d9c4 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:36:16.164752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08e1c87-3044-4d9f-ac1a-da8e5649eb14 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:36:16.165349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08e1c87-3044-4d9f-ac1a-da8e5649eb14 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:16.165622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08e1c87-3044-4d9f-ac1a-da8e5649eb14 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:16.165891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08e1c87-3044-4d9f-ac1a-da8e5649eb14 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:16.167653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d08e1c87-3044-4d9f-ac1a-da8e5649eb14 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:16.180233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08e1c87-3044-4d9f-ac1a-da8e5649eb14 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:36:16.181828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08e1c87-3044-4d9f-ac1a-da8e5649eb14 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:36:16.192981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1149/2298] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:36:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:36:16.891645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=63 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:36:26.188559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=18 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:36:34.976208 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9d5a2-7e01-46a4-80fb-bfa447fe8aa1 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:34.976728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9d5a2-7e01-46a4-80fb-bfa447fe8aa1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:34.976814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9d5a2-7e01-46a4-80fb-bfa447fe8aa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:34.998770 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-50c9d5a2-7e01-46a4-80fb-bfa447fe8aa1 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:36:35.000330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-50c9d5a2-7e01-46a4-80fb-bfa447fe8aa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:35.009174 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1150/2299] 10.222.0.16 () {58 vars in 967 bytes} [Tue Sep 13 15:36:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:36:35.065813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08e1c87-3044-4d9f-ac1a-da8e5649eb14 service nova] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:36:35.077969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08e1c87-3044-4d9f-ac1a-da8e5649eb14 service nova] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:36:35.083550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47f12eb-e770-4dbd-b679-21506de8f1ca service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:36:35.084154 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47f12eb-e770-4dbd-b679-21506de8f1ca service swift] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:35.084431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47f12eb-e770-4dbd-b679-21506de8f1ca service swift] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:35.084696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47f12eb-e770-4dbd-b679-21506de8f1ca service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:35.093084 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47f12eb-e770-4dbd-b679-21506de8f1ca service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:36:35.094722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47f12eb-e770-4dbd-b679-21506de8f1ca service swift] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:36:35.103558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1150/2300] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:36:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:36:38.496336 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc6b357-ba39-4c01-a060-42486b775f63 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:38.497237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc6b357-ba39-4c01-a060-42486b775f63 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:38.497540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc6b357-ba39-4c01-a060-42486b775f63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:38.523154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8bc6b357-ba39-4c01-a060-42486b775f63 None None] MFA Rules not processed for user `f48ce3ac32e741ef98e5025fd23bbaa0`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:36:38.524585 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8bc6b357-ba39-4c01-a060-42486b775f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:38.538072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1151/2301] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:36:38 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:36:39.708543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47f12eb-e770-4dbd-b679-21506de8f1ca service swift] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:36:39.722068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47f12eb-e770-4dbd-b679-21506de8f1ca service swift] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:36:39.727522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7e4f79-1316-40d4-abef-8f5283d30a26 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:36:39.728198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7e4f79-1316-40d4-abef-8f5283d30a26 service placement] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:39.728529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7e4f79-1316-40d4-abef-8f5283d30a26 service placement] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:39.728859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7e4f79-1316-40d4-abef-8f5283d30a26 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:39.730644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1c7e4f79-1316-40d4-abef-8f5283d30a26 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:39.743230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7e4f79-1316-40d4-abef-8f5283d30a26 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:36:39.745194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7e4f79-1316-40d4-abef-8f5283d30a26 service placement] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:36:39.756249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1151/2302] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:36:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:36:39.961154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25967fb8-e3f2-4d3c-96b4-5481a4eb9d7a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:39.961688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25967fb8-e3f2-4d3c-96b4-5481a4eb9d7a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:39.961688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25967fb8-e3f2-4d3c-96b4-5481a4eb9d7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:39.981869 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-25967fb8-e3f2-4d3c-96b4-5481a4eb9d7a None None] MFA Rules not processed for user `c02aec8271af4bf7aea728ff92fa555b`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:36:39.984883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-25967fb8-e3f2-4d3c-96b4-5481a4eb9d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:39.991502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1152/2303] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:36:39 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:36:44.554817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7e4f79-1316-40d4-abef-8f5283d30a26 service placement] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:36:44.558960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1c7e4f79-1316-40d4-abef-8f5283d30a26 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:44.625521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7e4f79-1316-40d4-abef-8f5283d30a26 service placement] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:36:44.630590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:36:44.631417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:44.631852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:44.632124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] PATH_INFO: `/v3/users/0815b4e3389240ada8d1c55941aca8cf` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:44.633258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] RBAC: Authorizing `identity:delete_user(user_id=0815b4e3389240ada8d1c55941aca8cf)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:36:44.635224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:36:44.685951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0815b4e3389240ada8d1c55941aca8cf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:44.686595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0815b4e3389240ada8d1c55941aca8cf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:44.693549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0815b4e3389240ada8d1c55941aca8cf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:44.694231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0815b4e3389240ada8d1c55941aca8cf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:44.704111 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0815b4e3389240ada8d1c55941aca8cf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:44.722017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1152/2304] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:36:44 2022] DELETE /identity/v3/users/0815b4e3389240ada8d1c55941aca8cf => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:36:44.894724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25967fb8-e3f2-4d3c-96b4-5481a4eb9d7a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:36:44.906227 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25967fb8-e3f2-4d3c-96b4-5481a4eb9d7a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:36:44.967442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:36:44.968122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:44.968399 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:44.968691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] PATH_INFO: `/v3/projects/74318a6f074f4d7ea084461473ef1a80` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:44.970043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74318a6f074f4d7ea084461473ef1a80)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:36:44.971986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:36:45.020880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74318a6f074f4d7ea084461473ef1a80', 'request_id': 'req-a6cda600-53ba-462e-a575-7e2449c6ec71'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.026374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74318a6f074f4d7ea084461473ef1a80', 'request_id': 'req-a6cda600-53ba-462e-a575-7e2449c6ec71'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.034837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74318a6f074f4d7ea084461473ef1a80', 'request_id': 'req-a6cda600-53ba-462e-a575-7e2449c6ec71'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.046924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] The token cache is being invalidate because project 74318a6f074f4d7ea084461473ef1a80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:36:45.047368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.048239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.049886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1153/2305] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:36:44 2022] DELETE /identity/v3/projects/74318a6f074f4d7ea084461473ef1a80 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:36:45.054151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:36:45.055645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:45.068863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09748df1-e12f-4f1d-bd76-ae7186e4846c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:36:45.118328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:36:45.129722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:36:45.130505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:45.130892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:45.131329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] PATH_INFO: `/v3/users/e8cbefad6fc24689af293becdfb0f990` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:45.132704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8cbefad6fc24689af293becdfb0f990)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:36:45.134662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:36:45.190546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8cbefad6fc24689af293becdfb0f990'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.191502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8cbefad6fc24689af293becdfb0f990'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.201793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8cbefad6fc24689af293becdfb0f990'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.202423 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8cbefad6fc24689af293becdfb0f990'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.214040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8cbefad6fc24689af293becdfb0f990'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.231385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1153/2306] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:36:45 2022] DELETE /identity/v3/users/e8cbefad6fc24689af293becdfb0f990 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:36:45.490452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:36:45.491968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:36:45.507944 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cda600-53ba-462e-a575-7e2449c6ec71 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:36:45.594785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:36:45.596603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:36:45.597081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:36:45.597483 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] PATH_INFO: `/v3/projects/cfaf2e58e0f14b95875b97b38a15dd63` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:36:45.598900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cfaf2e58e0f14b95875b97b38a15dd63)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:36:45.600905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:36:45.613573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:36:45.662594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cfaf2e58e0f14b95875b97b38a15dd63', 'request_id': 'req-16cb21a9-3a17-41e9-9339-0c6437eff8f1'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.668798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cfaf2e58e0f14b95875b97b38a15dd63', 'request_id': 'req-16cb21a9-3a17-41e9-9339-0c6437eff8f1'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.678928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cfaf2e58e0f14b95875b97b38a15dd63', 'request_id': 'req-16cb21a9-3a17-41e9-9339-0c6437eff8f1'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.692890 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] The token cache is being invalidate because project cfaf2e58e0f14b95875b97b38a15dd63 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:36:45.693239 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.693832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:36:45.695350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1154/2307] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:36:45 2022] DELETE /identity/v3/projects/cfaf2e58e0f14b95875b97b38a15dd63 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:36:55.228241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=29,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:36:55.689039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=19,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:03.095608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:03.097339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:03.158877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33aab78-4fc3-4d7b-b088-8b0bc2da826c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:03.214818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d0c15a-514f-4e2d-b4a6-aab498e0fef8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:03.215511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d0c15a-514f-4e2d-b4a6-aab498e0fef8 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:03.215824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d0c15a-514f-4e2d-b4a6-aab498e0fef8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:03.216093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d0c15a-514f-4e2d-b4a6-aab498e0fef8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:03.218055 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b9d0c15a-514f-4e2d-b4a6-aab498e0fef8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:03.282461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d0c15a-514f-4e2d-b4a6-aab498e0fef8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:03.284348 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d0c15a-514f-4e2d-b4a6-aab498e0fef8 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:03.293874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1154/2308] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:06.467759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:06.469307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:06.538644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d0c15a-514f-4e2d-b4a6-aab498e0fef8 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:06.541070 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b9d0c15a-514f-4e2d-b4a6-aab498e0fef8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:06.543068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cb21a9-3a17-41e9-9339-0c6437eff8f1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:06.548706 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c1181d-2591-4f19-a7fa-c06c19321ff0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:06.549321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c1181d-2591-4f19-a7fa-c06c19321ff0 service placement] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:06.549597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c1181d-2591-4f19-a7fa-c06c19321ff0 service placement] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:06.549866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c1181d-2591-4f19-a7fa-c06c19321ff0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:06.551816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-58c1181d-2591-4f19-a7fa-c06c19321ff0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:06.557228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d0c15a-514f-4e2d-b4a6-aab498e0fef8 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:06.562441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:06.563068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:06.563343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:06.563613 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] PATH_INFO: `/v3/users/97189d5c87c94be2bef7cff79c471c7d` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:06.564057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c1181d-2591-4f19-a7fa-c06c19321ff0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:06.564864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] RBAC: Authorizing `identity:delete_user(user_id=97189d5c87c94be2bef7cff79c471c7d)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:06.565807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c1181d-2591-4f19-a7fa-c06c19321ff0 service placement] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:06.566778 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:06.580225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1155/2309] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:06.621598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '97189d5c87c94be2bef7cff79c471c7d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:06.622283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '97189d5c87c94be2bef7cff79c471c7d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:06.631556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '97189d5c87c94be2bef7cff79c471c7d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:06.632292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '97189d5c87c94be2bef7cff79c471c7d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:06.636276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:06.644358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1903d569-b4fb-4e29-9c61-1c56275a3669 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '97189d5c87c94be2bef7cff79c471c7d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:06.664086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1155/2310] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:37:06 2022] DELETE /identity/v3/users/97189d5c87c94be2bef7cff79c471c7d => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:06.848852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c1181d-2591-4f19-a7fa-c06c19321ff0 service placement] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:06.862829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c1181d-2591-4f19-a7fa-c06c19321ff0 service placement] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:06.927233 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:06.927868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:06.928142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:06.928419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] PATH_INFO: `/v3/projects/07f8e7ebb2814be392842c2e17d55fca` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:06.929538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=07f8e7ebb2814be392842c2e17d55fca)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:06.931477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:06.982542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07f8e7ebb2814be392842c2e17d55fca', 'request_id': 'req-0abf7a09-9b14-472a-809d-fcaab1638a76'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:06.989215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07f8e7ebb2814be392842c2e17d55fca', 'request_id': 'req-0abf7a09-9b14-472a-809d-fcaab1638a76'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:06.998436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07f8e7ebb2814be392842c2e17d55fca', 'request_id': 'req-0abf7a09-9b14-472a-809d-fcaab1638a76'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:07.010347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] The token cache is being invalidate because project 07f8e7ebb2814be392842c2e17d55fca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:37:07.010635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:07.011337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0abf7a09-9b14-472a-809d-fcaab1638a76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:07.012907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1156/2311] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:37:06 2022] DELETE /identity/v3/projects/07f8e7ebb2814be392842c2e17d55fca => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:07.018310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe96e24-bb9f-4def-aabd-d32ff6759532 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:07.018529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe96e24-bb9f-4def-aabd-d32ff6759532 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:07.018796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe96e24-bb9f-4def-aabd-d32ff6759532 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:07.039996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-cfe96e24-bb9f-4def-aabd-d32ff6759532 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:07.041217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-cfe96e24-bb9f-4def-aabd-d32ff6759532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:07.105014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1156/2312] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:07.113995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91294c61-670a-4643-b6a3-93a0d795289a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:07.114387 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91294c61-670a-4643-b6a3-93a0d795289a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:07.114662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91294c61-670a-4643-b6a3-93a0d795289a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:07.135845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-91294c61-670a-4643-b6a3-93a0d795289a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:07.137300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-91294c61-670a-4643-b6a3-93a0d795289a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:07.146441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1157/2313] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:37:07.151658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe96e24-bb9f-4def-aabd-d32ff6759532 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:07.166904 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe96e24-bb9f-4def-aabd-d32ff6759532 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:07.174879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00fc7f4-1dd7-4527-a4ca-f37f294238c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:07.175475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00fc7f4-1dd7-4527-a4ca-f37f294238c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:07.175748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00fc7f4-1dd7-4527-a4ca-f37f294238c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:07.176011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00fc7f4-1dd7-4527-a4ca-f37f294238c0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:07.176656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00fc7f4-1dd7-4527-a4ca-f37f294238c0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:37:07.176944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00fc7f4-1dd7-4527-a4ca-f37f294238c0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:07.178579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00fc7f4-1dd7-4527-a4ca-f37f294238c0 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:07.190620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1157/2314] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:37:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:37:07.195125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91294c61-670a-4643-b6a3-93a0d795289a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:07.204657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91294c61-670a-4643-b6a3-93a0d795289a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:07.210676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e31609b-8300-4774-b019-9b64813837fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:07.211275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e31609b-8300-4774-b019-9b64813837fd admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:07.211568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e31609b-8300-4774-b019-9b64813837fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:07.211831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e31609b-8300-4774-b019-9b64813837fd admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:07.212465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e31609b-8300-4774-b019-9b64813837fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:07.214427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e31609b-8300-4774-b019-9b64813837fd admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:07.226432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:07.246959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1158/2315] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:37:07 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:07.251226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00fc7f4-1dd7-4527-a4ca-f37f294238c0 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:07.261560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00fc7f4-1dd7-4527-a4ca-f37f294238c0 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:07.322336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8989d1f4-31be-46a9-8faf-51b28cf76c0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:07.323046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8989d1f4-31be-46a9-8faf-51b28cf76c0a admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:07.323503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8989d1f4-31be-46a9-8faf-51b28cf76c0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:07.323809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8989d1f4-31be-46a9-8faf-51b28cf76c0a admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:07.324816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8989d1f4-31be-46a9-8faf-51b28cf76c0a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:07.327789 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8989d1f4-31be-46a9-8faf-51b28cf76c0a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:07.357117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1158/2316] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:37:07 2022] POST /identity/v3/users => generated 287 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:37:07.361527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e31609b-8300-4774-b019-9b64813837fd admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:07.372006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e31609b-8300-4774-b019-9b64813837fd admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:07.377481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2af204e-9076-4122-8ce8-914fe69ac08f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:07.378080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af204e-9076-4122-8ce8-914fe69ac08f admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:07.378379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af204e-9076-4122-8ce8-914fe69ac08f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:07.378643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af204e-9076-4122-8ce8-914fe69ac08f admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:07.379301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2af204e-9076-4122-8ce8-914fe69ac08f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:07.380910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2af204e-9076-4122-8ce8-914fe69ac08f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:07.391266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1159/2317] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:07.395537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8989d1f4-31be-46a9-8faf-51b28cf76c0a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:07.405802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8989d1f4-31be-46a9-8faf-51b28cf76c0a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:07.411745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9401a578-0917-459e-ad7a-f4686bd774df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:07.412344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401a578-0917-459e-ad7a-f4686bd774df admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:07.412636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401a578-0917-459e-ad7a-f4686bd774df admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:07.412885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9401a578-0917-459e-ad7a-f4686bd774df admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:07.413530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401a578-0917-459e-ad7a-f4686bd774df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:07.415175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9401a578-0917-459e-ad7a-f4686bd774df admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:07.426295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1159/2318] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:07.430543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2af204e-9076-4122-8ce8-914fe69ac08f admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:07.441038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2af204e-9076-4122-8ce8-914fe69ac08f admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:07.446271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93423ce-af32-4645-879c-19fa48d945da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:07.447438 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93423ce-af32-4645-879c-19fa48d945da admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:07.447438 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93423ce-af32-4645-879c-19fa48d945da admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:07.447662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93423ce-af32-4645-879c-19fa48d945da admin admin] PATH_INFO: `/v3/projects/113d200ad37e45fd85c40d4a9dd9a1f0/users/e3b8f96f086f4e319ca1cf8b8857d581/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:07.465535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93423ce-af32-4645-879c-19fa48d945da admin admin] RBAC: Authorizing `identity:create_grant(project_id=113d200ad37e45fd85c40d4a9dd9a1f0, user_id=e3b8f96f086f4e319ca1cf8b8857d581, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:07.468401 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93423ce-af32-4645-879c-19fa48d945da admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:07.484623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1160/2319] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:07 2022] PUT /identity/v3/projects/113d200ad37e45fd85c40d4a9dd9a1f0/users/e3b8f96f086f4e319ca1cf8b8857d581/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:07.495906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6980316b-1f56-4585-ad5c-e808a19b3adf None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:07.496191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6980316b-1f56-4585-ad5c-e808a19b3adf None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:07.496484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6980316b-1f56-4585-ad5c-e808a19b3adf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:07.526195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-6980316b-1f56-4585-ad5c-e808a19b3adf None None] MFA Rules not processed for user `e3b8f96f086f4e319ca1cf8b8857d581`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:07.527428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6980316b-1f56-4585-ad5c-e808a19b3adf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:07.566290 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:07.601675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1160/2320] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:07 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:37:07.613111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93423ce-af32-4645-879c-19fa48d945da admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:07.614939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e93423ce-af32-4645-879c-19fa48d945da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:07.691498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93423ce-af32-4645-879c-19fa48d945da admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:07.696932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bf79a1-54dd-418e-944d-6ac843efc0e9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:07.697902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf79a1-54dd-418e-944d-6ac843efc0e9 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:07.698222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf79a1-54dd-418e-944d-6ac843efc0e9 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:07.698491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf79a1-54dd-418e-944d-6ac843efc0e9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:07.707498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bf79a1-54dd-418e-944d-6ac843efc0e9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:07.709286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bf79a1-54dd-418e-944d-6ac843efc0e9 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:07.721558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1161/2321] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:07 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:10.456411 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6980316b-1f56-4585-ad5c-e808a19b3adf None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:10.546796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6980316b-1f56-4585-ad5c-e808a19b3adf None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:10.551646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:10.552275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:10.552551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:10.552814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] PATH_INFO: `/v3/users/e3b8f96f086f4e319ca1cf8b8857d581` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:10.553952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3b8f96f086f4e319ca1cf8b8857d581)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:10.555947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:10.607279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3b8f96f086f4e319ca1cf8b8857d581'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:10.607969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3b8f96f086f4e319ca1cf8b8857d581'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:10.615321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3b8f96f086f4e319ca1cf8b8857d581'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:10.615935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3b8f96f086f4e319ca1cf8b8857d581'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:10.625869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3b8f96f086f4e319ca1cf8b8857d581'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:10.644431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1161/2322] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:37:10 2022] DELETE /identity/v3/users/e3b8f96f086f4e319ca1cf8b8857d581 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:10.649102 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468a6f2e-91df-49c7-b093-8e4f57ec89b0 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:10.649598 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468a6f2e-91df-49c7-b093-8e4f57ec89b0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:10.649714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468a6f2e-91df-49c7-b093-8e4f57ec89b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:10.668513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-468a6f2e-91df-49c7-b093-8e4f57ec89b0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:10.669620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-468a6f2e-91df-49c7-b093-8e4f57ec89b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:10.730622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1162/2323] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:10.740569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:10.742208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:10.813672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dfc820-b345-48db-9202-6422656651e1 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:10.818075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e711cbe0-4753-4a92-8fc5-c294436e843d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:10.818722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e711cbe0-4753-4a92-8fc5-c294436e843d service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:10.819005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e711cbe0-4753-4a92-8fc5-c294436e843d service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:10.819274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e711cbe0-4753-4a92-8fc5-c294436e843d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:10.826891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:10.829254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e711cbe0-4753-4a92-8fc5-c294436e843d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:10.831408 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e711cbe0-4753-4a92-8fc5-c294436e843d service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:10.842611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1162/2324] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:11.154572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468a6f2e-91df-49c7-b093-8e4f57ec89b0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:11.156103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-468a6f2e-91df-49c7-b093-8e4f57ec89b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:11.169097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468a6f2e-91df-49c7-b093-8e4f57ec89b0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:11.174505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:11.175170 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.175578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.175863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] PATH_INFO: `/v3/projects/113d200ad37e45fd85c40d4a9dd9a1f0` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.177514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] RBAC: Authorizing `identity:delete_project(project_id=113d200ad37e45fd85c40d4a9dd9a1f0)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:11.178975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:11.219812 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:11.229320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '113d200ad37e45fd85c40d4a9dd9a1f0', 'request_id': 'req-3e58d062-8939-4459-a734-345f5b91424f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:11.234836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '113d200ad37e45fd85c40d4a9dd9a1f0', 'request_id': 'req-3e58d062-8939-4459-a734-345f5b91424f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:11.244733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '113d200ad37e45fd85c40d4a9dd9a1f0', 'request_id': 'req-3e58d062-8939-4459-a734-345f5b91424f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:11.257126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] The token cache is being invalidate because project 113d200ad37e45fd85c40d4a9dd9a1f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:37:11.257415 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:11.258149 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3e58d062-8939-4459-a734-345f5b91424f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:11.259762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1163/2325] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:37:11 2022] DELETE /identity/v3/projects/113d200ad37e45fd85c40d4a9dd9a1f0 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:11.265039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c84970-9f72-469b-b714-c23f009e5144 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.265332 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c84970-9f72-469b-b714-c23f009e5144 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.265604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c84970-9f72-469b-b714-c23f009e5144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.285927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-21c84970-9f72-469b-b714-c23f009e5144 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:11.287175 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-21c84970-9f72-469b-b714-c23f009e5144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:11.353058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1163/2326] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:11.361269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d763563-fccb-48ec-84ae-69db98c3ab00 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.361560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d763563-fccb-48ec-84ae-69db98c3ab00 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.361953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d763563-fccb-48ec-84ae-69db98c3ab00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.383128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0d763563-fccb-48ec-84ae-69db98c3ab00 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:11.384381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0d763563-fccb-48ec-84ae-69db98c3ab00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:11.394140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1164/2327] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:11.398518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c84970-9f72-469b-b714-c23f009e5144 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:11.408531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c84970-9f72-469b-b714-c23f009e5144 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:11.414459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2ebff-1631-4591-8f1c-94096aba9275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:11.415058 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2ebff-1631-4591-8f1c-94096aba9275 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.415328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2ebff-1631-4591-8f1c-94096aba9275 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.415595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2ebff-1631-4591-8f1c-94096aba9275 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.416232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa2ebff-1631-4591-8f1c-94096aba9275 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:37:11.416529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa2ebff-1631-4591-8f1c-94096aba9275 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:11.418141 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa2ebff-1631-4591-8f1c-94096aba9275 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:11.429952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1164/2328] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:37:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:37:11.434348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d763563-fccb-48ec-84ae-69db98c3ab00 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:11.444116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d763563-fccb-48ec-84ae-69db98c3ab00 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:11.450283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09805bce-72a3-400f-8e0b-1a572e70cfa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:11.450924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09805bce-72a3-400f-8e0b-1a572e70cfa0 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.451200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09805bce-72a3-400f-8e0b-1a572e70cfa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.451466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09805bce-72a3-400f-8e0b-1a572e70cfa0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.452156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09805bce-72a3-400f-8e0b-1a572e70cfa0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:11.454065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09805bce-72a3-400f-8e0b-1a572e70cfa0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:11.485578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1165/2329] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:37:11 2022] POST /identity/v3/projects => generated 405 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:11.489967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2ebff-1631-4591-8f1c-94096aba9275 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:11.501894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2ebff-1631-4591-8f1c-94096aba9275 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:11.562283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372d4a61-a6c2-4fb6-a3eb-44b88ec5d4fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:11.562891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d4a61-a6c2-4fb6-a3eb-44b88ec5d4fd admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.563163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d4a61-a6c2-4fb6-a3eb-44b88ec5d4fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.563431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372d4a61-a6c2-4fb6-a3eb-44b88ec5d4fd admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.564072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372d4a61-a6c2-4fb6-a3eb-44b88ec5d4fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:11.566157 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372d4a61-a6c2-4fb6-a3eb-44b88ec5d4fd admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:11.590800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1165/2330] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:37:11 2022] POST /identity/v3/users => generated 303 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:11.595108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09805bce-72a3-400f-8e0b-1a572e70cfa0 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:11.606117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09805bce-72a3-400f-8e0b-1a572e70cfa0 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:11.611025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b77925-de02-4e78-9887-9d4e93eff958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:11.611720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b77925-de02-4e78-9887-9d4e93eff958 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.611995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b77925-de02-4e78-9887-9d4e93eff958 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.612409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b77925-de02-4e78-9887-9d4e93eff958 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.613054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b77925-de02-4e78-9887-9d4e93eff958 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:11.614732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b77925-de02-4e78-9887-9d4e93eff958 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:11.625244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1166/2331] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:37:11.629568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372d4a61-a6c2-4fb6-a3eb-44b88ec5d4fd admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:11.640630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372d4a61-a6c2-4fb6-a3eb-44b88ec5d4fd admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:11.646043 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297657f7-67d6-4e80-a15b-973160d9079d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:11.646427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297657f7-67d6-4e80-a15b-973160d9079d admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.646699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297657f7-67d6-4e80-a15b-973160d9079d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.646959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297657f7-67d6-4e80-a15b-973160d9079d admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.647607 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297657f7-67d6-4e80-a15b-973160d9079d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:11.649248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297657f7-67d6-4e80-a15b-973160d9079d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:11.661080 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1166/2332] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:11.665430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b77925-de02-4e78-9887-9d4e93eff958 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:11.675738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b77925-de02-4e78-9887-9d4e93eff958 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:11.680360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e0d1ef-500f-400e-baa8-5c68dfe4966e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:11.681008 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e0d1ef-500f-400e-baa8-5c68dfe4966e admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.681280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e0d1ef-500f-400e-baa8-5c68dfe4966e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.681547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e0d1ef-500f-400e-baa8-5c68dfe4966e admin admin] PATH_INFO: `/v3/projects/f13bf78ebba1443cb61633bf1fd8093b/users/dede592f05dc431b9003ae193fa0d502/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.696762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e0d1ef-500f-400e-baa8-5c68dfe4966e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f13bf78ebba1443cb61633bf1fd8093b, user_id=dede592f05dc431b9003ae193fa0d502, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:11.699425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e0d1ef-500f-400e-baa8-5c68dfe4966e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:11.721232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1167/2333] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:11 2022] PUT /identity/v3/projects/f13bf78ebba1443cb61633bf1fd8093b/users/dede592f05dc431b9003ae193fa0d502/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:11.727262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3baad8-4c2d-4d87-af01-0a5cbc4faccb None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.727619 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3baad8-4c2d-4d87-af01-0a5cbc4faccb None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.728026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3baad8-4c2d-4d87-af01-0a5cbc4faccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.757132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-0a3baad8-4c2d-4d87-af01-0a5cbc4faccb None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:11.758870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0a3baad8-4c2d-4d87-af01-0a5cbc4faccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:11.833630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1167/2334] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:11.845261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e0d1ef-500f-400e-baa8-5c68dfe4966e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:11.847717 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-17e0d1ef-500f-400e-baa8-5c68dfe4966e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:11.908749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e0d1ef-500f-400e-baa8-5c68dfe4966e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:11.912852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12644445-1a33-400f-b651-5ff6b9a17b2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:11.913455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12644445-1a33-400f-b651-5ff6b9a17b2b service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:11.913730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12644445-1a33-400f-b651-5ff6b9a17b2b service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:11.913996 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12644445-1a33-400f-b651-5ff6b9a17b2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:11.923381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12644445-1a33-400f-b651-5ff6b9a17b2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:11.925407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12644445-1a33-400f-b651-5ff6b9a17b2b service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:11.934866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1168/2335] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:12.576437 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ab39ef-2108-4735-9348-9331887bb94e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:12.577020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ab39ef-2108-4735-9348-9331887bb94e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:12.577020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ab39ef-2108-4735-9348-9331887bb94e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:12.584811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:12.607161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-65ab39ef-2108-4735-9348-9331887bb94e None None] MFA Rules not processed for user `dede592f05dc431b9003ae193fa0d502`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:12.608482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-65ab39ef-2108-4735-9348-9331887bb94e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:12.684928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1168/2336] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:12 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:12.689547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12644445-1a33-400f-b651-5ff6b9a17b2b service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:12.699623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12644445-1a33-400f-b651-5ff6b9a17b2b service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:12.704219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e36c973-7ded-413b-986e-628a7fd63be1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:12.704822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e36c973-7ded-413b-986e-628a7fd63be1 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:12.705098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e36c973-7ded-413b-986e-628a7fd63be1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:12.705724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e36c973-7ded-413b-986e-628a7fd63be1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:12.706386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e36c973-7ded-413b-986e-628a7fd63be1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:12.708267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e36c973-7ded-413b-986e-628a7fd63be1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:12.736655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1169/2337] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:37:12 2022] POST /identity/v3/projects => generated 405 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:12.740754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ab39ef-2108-4735-9348-9331887bb94e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:12.751758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ab39ef-2108-4735-9348-9331887bb94e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:12.812282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edab31b3-2b4a-490e-84ec-5bb74552c996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:12.812888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edab31b3-2b4a-490e-84ec-5bb74552c996 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:12.813160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edab31b3-2b4a-490e-84ec-5bb74552c996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:12.813427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edab31b3-2b4a-490e-84ec-5bb74552c996 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:12.814034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edab31b3-2b4a-490e-84ec-5bb74552c996 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:12.816084 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edab31b3-2b4a-490e-84ec-5bb74552c996 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:12.841538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1169/2338] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:37:12 2022] POST /identity/v3/users => generated 309 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:12.845781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e36c973-7ded-413b-986e-628a7fd63be1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:12.856569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e36c973-7ded-413b-986e-628a7fd63be1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:12.861407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da6faea-cfa6-4060-8321-fe900f372298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:12.862144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da6faea-cfa6-4060-8321-fe900f372298 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:12.862426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da6faea-cfa6-4060-8321-fe900f372298 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:12.862699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da6faea-cfa6-4060-8321-fe900f372298 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:12.863343 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da6faea-cfa6-4060-8321-fe900f372298 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:12.864947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da6faea-cfa6-4060-8321-fe900f372298 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:12.875494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1170/2339] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:12.880472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edab31b3-2b4a-490e-84ec-5bb74552c996 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:12.890975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edab31b3-2b4a-490e-84ec-5bb74552c996 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:12.897691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73054513-2a38-401c-af54-c2f91e81e4ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:12.898373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73054513-2a38-401c-af54-c2f91e81e4ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:12.898652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73054513-2a38-401c-af54-c2f91e81e4ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:12.898921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73054513-2a38-401c-af54-c2f91e81e4ac admin admin] PATH_INFO: `/v3/projects/3b414ca37b204ba8a6f756aa2d852d37/users/2c559ce0347e4f96a0cd9b323faa18f0/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:12.914871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73054513-2a38-401c-af54-c2f91e81e4ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b414ca37b204ba8a6f756aa2d852d37, user_id=2c559ce0347e4f96a0cd9b323faa18f0, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:12.917542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73054513-2a38-401c-af54-c2f91e81e4ac admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:12.932764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1170/2340] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:12 2022] PUT /identity/v3/projects/3b414ca37b204ba8a6f756aa2d852d37/users/2c559ce0347e4f96a0cd9b323faa18f0/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:12.936924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da6faea-cfa6-4060-8321-fe900f372298 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:12.947395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da6faea-cfa6-4060-8321-fe900f372298 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:12.978326 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:13.008797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0899a4-3abf-4595-ab69-78eb39bd6528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:13.009424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0899a4-3abf-4595-ab69-78eb39bd6528 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:13.009702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0899a4-3abf-4595-ab69-78eb39bd6528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:13.009968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0899a4-3abf-4595-ab69-78eb39bd6528 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:13.010641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0899a4-3abf-4595-ab69-78eb39bd6528 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:13.012275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0899a4-3abf-4595-ab69-78eb39bd6528 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:13.022683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1171/2341] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:12 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:13.027060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73054513-2a38-401c-af54-c2f91e81e4ac admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:13.037927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73054513-2a38-401c-af54-c2f91e81e4ac admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:13.042818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a76b50c-f427-4e07-978f-f8ad54cbb107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:13.043463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a76b50c-f427-4e07-978f-f8ad54cbb107 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:13.043738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a76b50c-f427-4e07-978f-f8ad54cbb107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:13.044010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a76b50c-f427-4e07-978f-f8ad54cbb107 admin admin] PATH_INFO: `/v3/projects/3b414ca37b204ba8a6f756aa2d852d37/users/2c559ce0347e4f96a0cd9b323faa18f0/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:13.045924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a76b50c-f427-4e07-978f-f8ad54cbb107 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b414ca37b204ba8a6f756aa2d852d37, user_id=2c559ce0347e4f96a0cd9b323faa18f0, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:13.048621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a76b50c-f427-4e07-978f-f8ad54cbb107 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:13.059069 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1171/2342] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:13 2022] PUT /identity/v3/projects/3b414ca37b204ba8a6f756aa2d852d37/users/2c559ce0347e4f96a0cd9b323faa18f0/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:37:13.710232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fa7fba-66fe-4ed8-bd21-00692138a4a0 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:13.710779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fa7fba-66fe-4ed8-bd21-00692138a4a0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:13.710894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fa7fba-66fe-4ed8-bd21-00692138a4a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:13.740251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-92fa7fba-66fe-4ed8-bd21-00692138a4a0 None None] MFA Rules not processed for user `2c559ce0347e4f96a0cd9b323faa18f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:13.741475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-92fa7fba-66fe-4ed8-bd21-00692138a4a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:13.812947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1172/2343] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:13 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:14.870850 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a76b50c-f427-4e07-978f-f8ad54cbb107 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:14.919600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:14.942902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a76b50c-f427-4e07-978f-f8ad54cbb107 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:14.947759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:14.948383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:14.948657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:14.948928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] PATH_INFO: `/v3/users/dede592f05dc431b9003ae193fa0d502` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:14.950048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dede592f05dc431b9003ae193fa0d502)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:14.952018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:15.004319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dede592f05dc431b9003ae193fa0d502'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.004906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dede592f05dc431b9003ae193fa0d502'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.011723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dede592f05dc431b9003ae193fa0d502'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.012343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dede592f05dc431b9003ae193fa0d502'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.020659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dede592f05dc431b9003ae193fa0d502'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.039284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1172/2344] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:37:14 2022] DELETE /identity/v3/users/dede592f05dc431b9003ae193fa0d502 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:15.220550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fa7fba-66fe-4ed8-bd21-00692138a4a0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:15.233362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fa7fba-66fe-4ed8-bd21-00692138a4a0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:15.291988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:15.292548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:15.292828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:15.293098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] PATH_INFO: `/v3/projects/f13bf78ebba1443cb61633bf1fd8093b` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:15.294298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f13bf78ebba1443cb61633bf1fd8093b)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:15.296222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:15.349168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f13bf78ebba1443cb61633bf1fd8093b', 'request_id': 'req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.353014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:15.355318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f13bf78ebba1443cb61633bf1fd8093b', 'request_id': 'req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.365006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f13bf78ebba1443cb61633bf1fd8093b', 'request_id': 'req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.376391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] The token cache is being invalidate because project f13bf78ebba1443cb61633bf1fd8093b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:37:15.376670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.377456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.379155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1173/2345] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:37:15 2022] DELETE /identity/v3/projects/f13bf78ebba1443cb61633bf1fd8093b => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:15.383515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:15.385004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:15.399036 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be845ff-3063-43ff-8622-36eb9ade7e96 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:15.460053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:15.460751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:15.461019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:15.461307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] PATH_INFO: `/v3/users/2c559ce0347e4f96a0cd9b323faa18f0` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:15.462519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c559ce0347e4f96a0cd9b323faa18f0)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:15.464508 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:15.519798 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c559ce0347e4f96a0cd9b323faa18f0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.520438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c559ce0347e4f96a0cd9b323faa18f0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.527164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c559ce0347e4f96a0cd9b323faa18f0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.527764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c559ce0347e4f96a0cd9b323faa18f0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.536505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e4d47d01-4682-424a-ae50-d9ffb97b30cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c559ce0347e4f96a0cd9b323faa18f0'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.539009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:15.559595 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1173/2346] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:37:15 2022] DELETE /identity/v3/users/2c559ce0347e4f96a0cd9b323faa18f0 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:15.740708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:15.742327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:15.760643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9165f8-caa5-40cc-95a2-13cfb532dfe4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:15.822955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:15.823517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:15.823791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:15.824059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] PATH_INFO: `/v3/projects/3b414ca37b204ba8a6f756aa2d852d37` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:15.825298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b414ca37b204ba8a6f756aa2d852d37)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:15.827219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:15.876729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b414ca37b204ba8a6f756aa2d852d37', 'request_id': 'req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.882861 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b414ca37b204ba8a6f756aa2d852d37', 'request_id': 'req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.892634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b414ca37b204ba8a6f756aa2d852d37', 'request_id': 'req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.905205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] The token cache is being invalidate because project 3b414ca37b204ba8a6f756aa2d852d37 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:37:15.905493 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.906217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1a3c39c6-e7d0-4743-9fb9-43f02644d1c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:15.907777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1174/2347] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:37:15 2022] DELETE /identity/v3/projects/3b414ca37b204ba8a6f756aa2d852d37 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:15.912881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fca2c06-8909-419a-8c72-557d73b4c37a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:15.913436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fca2c06-8909-419a-8c72-557d73b4c37a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:15.913436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fca2c06-8909-419a-8c72-557d73b4c37a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:15.933907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3fca2c06-8909-419a-8c72-557d73b4c37a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:15.935184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3fca2c06-8909-419a-8c72-557d73b4c37a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:15.994986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1174/2348] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:16.003079 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350244c5-bbdf-4163-b3c2-a5ddd0262890 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.003360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350244c5-bbdf-4163-b3c2-a5ddd0262890 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.003629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350244c5-bbdf-4163-b3c2-a5ddd0262890 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.025078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-350244c5-bbdf-4163-b3c2-a5ddd0262890 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:16.026314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-350244c5-bbdf-4163-b3c2-a5ddd0262890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:16.036254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1175/2349] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:16.040650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fca2c06-8909-419a-8c72-557d73b4c37a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.050769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fca2c06-8909-419a-8c72-557d73b4c37a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.055817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaedb94c-27c8-48c0-8977-406537b0b71f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.056421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaedb94c-27c8-48c0-8977-406537b0b71f admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.056699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaedb94c-27c8-48c0-8977-406537b0b71f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.056966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaedb94c-27c8-48c0-8977-406537b0b71f admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.057607 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaedb94c-27c8-48c0-8977-406537b0b71f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:37:16.057907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaedb94c-27c8-48c0-8977-406537b0b71f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.059562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaedb94c-27c8-48c0-8977-406537b0b71f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.072479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1175/2350] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:37:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:37:16.077755 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350244c5-bbdf-4163-b3c2-a5ddd0262890 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.089311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350244c5-bbdf-4163-b3c2-a5ddd0262890 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.094662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7ba4a8-d890-44b9-88cf-a8c91c0cd924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.095547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ba4a8-d890-44b9-88cf-a8c91c0cd924 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.095831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ba4a8-d890-44b9-88cf-a8c91c0cd924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.096087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ba4a8-d890-44b9-88cf-a8c91c0cd924 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.096887 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7ba4a8-d890-44b9-88cf-a8c91c0cd924 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.098823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7ba4a8-d890-44b9-88cf-a8c91c0cd924 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.128645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1176/2351] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:37:16 2022] POST /identity/v3/projects => generated 367 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:16.132908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaedb94c-27c8-48c0-8977-406537b0b71f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.142646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaedb94c-27c8-48c0-8977-406537b0b71f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.201634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f385c73-590e-459f-8da3-6fea057368a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.202282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f385c73-590e-459f-8da3-6fea057368a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.202558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f385c73-590e-459f-8da3-6fea057368a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.202826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f385c73-590e-459f-8da3-6fea057368a3 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.203456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f385c73-590e-459f-8da3-6fea057368a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.205744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f385c73-590e-459f-8da3-6fea057368a3 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.229767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1176/2352] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:37:16 2022] POST /identity/v3/users => generated 284 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:16.236519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7ba4a8-d890-44b9-88cf-a8c91c0cd924 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.252983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7ba4a8-d890-44b9-88cf-a8c91c0cd924 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.258776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc328136-7b54-45f0-b289-734191072ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.259546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc328136-7b54-45f0-b289-734191072ac5 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.259843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc328136-7b54-45f0-b289-734191072ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.260130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc328136-7b54-45f0-b289-734191072ac5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.260769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc328136-7b54-45f0-b289-734191072ac5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.263160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc328136-7b54-45f0-b289-734191072ac5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.273820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1177/2353] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:16.277974 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f385c73-590e-459f-8da3-6fea057368a3 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.288574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f385c73-590e-459f-8da3-6fea057368a3 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.293488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74269129-304e-46fa-8e67-b52a231eeb05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.294118 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74269129-304e-46fa-8e67-b52a231eeb05 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.294366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74269129-304e-46fa-8e67-b52a231eeb05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.294669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74269129-304e-46fa-8e67-b52a231eeb05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.295313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74269129-304e-46fa-8e67-b52a231eeb05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.296954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74269129-304e-46fa-8e67-b52a231eeb05 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.307977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1177/2354] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:16.312581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc328136-7b54-45f0-b289-734191072ac5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.323350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc328136-7b54-45f0-b289-734191072ac5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.329167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a80738-2f27-45ad-a6fb-6f96b96ef76e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.329760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a80738-2f27-45ad-a6fb-6f96b96ef76e admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.330027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a80738-2f27-45ad-a6fb-6f96b96ef76e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.330694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a80738-2f27-45ad-a6fb-6f96b96ef76e admin admin] PATH_INFO: `/v3/projects/657d813e7f0c4774ba8c31adb70c7d82/users/14fc9879b7684876ab182f22715ce1b6/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.346145 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a80738-2f27-45ad-a6fb-6f96b96ef76e admin admin] RBAC: Authorizing `identity:create_grant(project_id=657d813e7f0c4774ba8c31adb70c7d82, user_id=14fc9879b7684876ab182f22715ce1b6, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.348862 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a80738-2f27-45ad-a6fb-6f96b96ef76e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.364725 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1178/2355] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:16 2022] PUT /identity/v3/projects/657d813e7f0c4774ba8c31adb70c7d82/users/14fc9879b7684876ab182f22715ce1b6/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:16.376181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b022030-f777-4674-8dd9-5960dafa8e83 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.376482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b022030-f777-4674-8dd9-5960dafa8e83 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.376862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b022030-f777-4674-8dd9-5960dafa8e83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.406195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-6b022030-f777-4674-8dd9-5960dafa8e83 None None] MFA Rules not processed for user `14fc9879b7684876ab182f22715ce1b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:16.407426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6b022030-f777-4674-8dd9-5960dafa8e83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:16.447146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:16.477038 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1178/2356] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:16 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:16.481656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a80738-2f27-45ad-a6fb-6f96b96ef76e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.492182 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:16.548006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a80738-2f27-45ad-a6fb-6f96b96ef76e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.553640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c33b86-57f7-42f6-82bb-9385b8103705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.554270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c33b86-57f7-42f6-82bb-9385b8103705 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.554549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c33b86-57f7-42f6-82bb-9385b8103705 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.554822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c33b86-57f7-42f6-82bb-9385b8103705 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.555459 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c33b86-57f7-42f6-82bb-9385b8103705 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.557350 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c33b86-57f7-42f6-82bb-9385b8103705 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.586456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1179/2357] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:37:16 2022] POST /identity/v3/projects => generated 367 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:16.590570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b022030-f777-4674-8dd9-5960dafa8e83 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.601363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b022030-f777-4674-8dd9-5960dafa8e83 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.662262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef24015-a7c8-440a-af6d-e8d4d72e008d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.662881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef24015-a7c8-440a-af6d-e8d4d72e008d admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.663156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef24015-a7c8-440a-af6d-e8d4d72e008d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.663422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef24015-a7c8-440a-af6d-e8d4d72e008d admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.664046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef24015-a7c8-440a-af6d-e8d4d72e008d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.665979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef24015-a7c8-440a-af6d-e8d4d72e008d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.689842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1179/2358] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:37:16 2022] POST /identity/v3/users => generated 290 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:16.694542 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c33b86-57f7-42f6-82bb-9385b8103705 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.705467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c33b86-57f7-42f6-82bb-9385b8103705 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.710431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376f233d-32fd-4dc2-ae3a-f5cfff08babb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.711038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376f233d-32fd-4dc2-ae3a-f5cfff08babb admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.711316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376f233d-32fd-4dc2-ae3a-f5cfff08babb admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.711583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376f233d-32fd-4dc2-ae3a-f5cfff08babb admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.712232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376f233d-32fd-4dc2-ae3a-f5cfff08babb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.713839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376f233d-32fd-4dc2-ae3a-f5cfff08babb admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.724484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1180/2359] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:16.728628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef24015-a7c8-440a-af6d-e8d4d72e008d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.738964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef24015-a7c8-440a-af6d-e8d4d72e008d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.744165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6aabbb7-48d8-4ea8-978b-f49e02032f77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.744813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6aabbb7-48d8-4ea8-978b-f49e02032f77 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.745095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6aabbb7-48d8-4ea8-978b-f49e02032f77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.745380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6aabbb7-48d8-4ea8-978b-f49e02032f77 admin admin] PATH_INFO: `/v3/projects/d4513e8c02d44ea1a6b0baf406358e90/users/ba84410e5f324092aaa9c3265b788d55/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.760686 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6aabbb7-48d8-4ea8-978b-f49e02032f77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4513e8c02d44ea1a6b0baf406358e90, user_id=ba84410e5f324092aaa9c3265b788d55, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.763716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6aabbb7-48d8-4ea8-978b-f49e02032f77 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.778759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1180/2360] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:16 2022] PUT /identity/v3/projects/d4513e8c02d44ea1a6b0baf406358e90/users/ba84410e5f324092aaa9c3265b788d55/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:16.782828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376f233d-32fd-4dc2-ae3a-f5cfff08babb admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.792934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376f233d-32fd-4dc2-ae3a-f5cfff08babb admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.850369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75263c22-d28c-4a50-9214-9e12de4010ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.850889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75263c22-d28c-4a50-9214-9e12de4010ce admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.851165 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75263c22-d28c-4a50-9214-9e12de4010ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.851437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75263c22-d28c-4a50-9214-9e12de4010ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.852098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75263c22-d28c-4a50-9214-9e12de4010ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.853690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75263c22-d28c-4a50-9214-9e12de4010ce admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.864417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1181/2361] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:16.871145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6aabbb7-48d8-4ea8-978b-f49e02032f77 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:16.883376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6aabbb7-48d8-4ea8-978b-f49e02032f77 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:16.890775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44da09e1-1af3-4de2-b5e3-c3ee0f8cd656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:16.891781 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44da09e1-1af3-4de2-b5e3-c3ee0f8cd656 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.892054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44da09e1-1af3-4de2-b5e3-c3ee0f8cd656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.892321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44da09e1-1af3-4de2-b5e3-c3ee0f8cd656 admin admin] PATH_INFO: `/v3/projects/d4513e8c02d44ea1a6b0baf406358e90/users/ba84410e5f324092aaa9c3265b788d55/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.894505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44da09e1-1af3-4de2-b5e3-c3ee0f8cd656 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4513e8c02d44ea1a6b0baf406358e90, user_id=ba84410e5f324092aaa9c3265b788d55, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:16.897187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44da09e1-1af3-4de2-b5e3-c3ee0f8cd656 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:16.907684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1181/2362] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:16 2022] PUT /identity/v3/projects/d4513e8c02d44ea1a6b0baf406358e90/users/ba84410e5f324092aaa9c3265b788d55/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:16.920286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb3be2-384c-407f-87c5-ff6360c247b8 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:16.920654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb3be2-384c-407f-87c5-ff6360c247b8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:16.921027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb3be2-384c-407f-87c5-ff6360c247b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:16.958346 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b8bb3be2-384c-407f-87c5-ff6360c247b8 None None] MFA Rules not processed for user `ba84410e5f324092aaa9c3265b788d55`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:16.959880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b8bb3be2-384c-407f-87c5-ff6360c247b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:17.031926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1182/2363] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:16 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:37:17.045775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44da09e1-1af3-4de2-b5e3-c3ee0f8cd656 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:17.047889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-44da09e1-1af3-4de2-b5e3-c3ee0f8cd656 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:17.113587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44da09e1-1af3-4de2-b5e3-c3ee0f8cd656 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:17.117639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b364fcd9-11df-43f9-92a2-46fe75bad8e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:17.118275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b364fcd9-11df-43f9-92a2-46fe75bad8e3 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:17.118551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b364fcd9-11df-43f9-92a2-46fe75bad8e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:17.118826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b364fcd9-11df-43f9-92a2-46fe75bad8e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:17.128684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b364fcd9-11df-43f9-92a2-46fe75bad8e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:17.130142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b364fcd9-11df-43f9-92a2-46fe75bad8e3 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:17.139997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1182/2364] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:17 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:18.629049 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bb3be2-384c-407f-87c5-ff6360c247b8 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:18.636461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:18.697117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bb3be2-384c-407f-87c5-ff6360c247b8 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:18.701644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:18.702311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:18.702587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:18.702854 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] PATH_INFO: `/v3/users/14fc9879b7684876ab182f22715ce1b6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:18.703965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=14fc9879b7684876ab182f22715ce1b6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:18.706356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:18.760818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14fc9879b7684876ab182f22715ce1b6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:18.769409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14fc9879b7684876ab182f22715ce1b6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:18.770243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14fc9879b7684876ab182f22715ce1b6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:18.771059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14fc9879b7684876ab182f22715ce1b6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:18.779621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14fc9879b7684876ab182f22715ce1b6'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:18.799335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1183/2365] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:37:18 2022] DELETE /identity/v3/users/14fc9879b7684876ab182f22715ce1b6 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:18.804170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff217b13-ce12-4dca-b614-7bd98fa57106 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:18.804654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff217b13-ce12-4dca-b614-7bd98fa57106 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:18.804654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff217b13-ce12-4dca-b614-7bd98fa57106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:18.824213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-ff217b13-ce12-4dca-b614-7bd98fa57106 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:18.825425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ff217b13-ce12-4dca-b614-7bd98fa57106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:18.856743 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:18.889032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1183/2366] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:18.899558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:18.901090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:18.958768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1581118-1e83-4fbd-8c75-a80b26bafc91 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:18.962974 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bee132-a3f9-4220-bf1e-6d6d53aa4a45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:18.963602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bee132-a3f9-4220-bf1e-6d6d53aa4a45 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:18.963839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bee132-a3f9-4220-bf1e-6d6d53aa4a45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:18.964128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bee132-a3f9-4220-bf1e-6d6d53aa4a45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:18.978732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bee132-a3f9-4220-bf1e-6d6d53aa4a45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:18.981043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bee132-a3f9-4220-bf1e-6d6d53aa4a45 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:18.993173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1184/2367] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:19.280562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff217b13-ce12-4dca-b614-7bd98fa57106 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:19.282033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ff217b13-ce12-4dca-b614-7bd98fa57106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:19.295366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff217b13-ce12-4dca-b614-7bd98fa57106 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:19.301542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:19.301820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:19.302092 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:19.302407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] PATH_INFO: `/v3/projects/657d813e7f0c4774ba8c31adb70c7d82` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:19.303653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] RBAC: Authorizing `identity:delete_project(project_id=657d813e7f0c4774ba8c31adb70c7d82)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:19.305688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:19.354671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '657d813e7f0c4774ba8c31adb70c7d82', 'request_id': 'req-4add5deb-af9f-41ec-bcca-f76a0e270224'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.360318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '657d813e7f0c4774ba8c31adb70c7d82', 'request_id': 'req-4add5deb-af9f-41ec-bcca-f76a0e270224'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.370064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '657d813e7f0c4774ba8c31adb70c7d82', 'request_id': 'req-4add5deb-af9f-41ec-bcca-f76a0e270224'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.381346 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] The token cache is being invalidate because project 657d813e7f0c4774ba8c31adb70c7d82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:37:19.381646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.382323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.383797 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1184/2368] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:37:19 2022] DELETE /identity/v3/projects/657d813e7f0c4774ba8c31adb70c7d82 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:19.387879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bee132-a3f9-4220-bf1e-6d6d53aa4a45 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:19.398400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bee132-a3f9-4220-bf1e-6d6d53aa4a45 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:19.424351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:19.459435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:19.460047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:19.460309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:19.460581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] PATH_INFO: `/v3/users/ba84410e5f324092aaa9c3265b788d55` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:19.461730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba84410e5f324092aaa9c3265b788d55)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:19.463657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:19.521474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba84410e5f324092aaa9c3265b788d55'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.530945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba84410e5f324092aaa9c3265b788d55'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.531658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba84410e5f324092aaa9c3265b788d55'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.532316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba84410e5f324092aaa9c3265b788d55'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.539762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a56a7452-e84d-412c-8c73-137c8f9527be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba84410e5f324092aaa9c3265b788d55'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.557103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1185/2369] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:37:19 2022] DELETE /identity/v3/users/ba84410e5f324092aaa9c3265b788d55 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:19.843444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:19.844911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:19.858003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4add5deb-af9f-41ec-bcca-f76a0e270224 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:19.917843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:19.918519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:19.918792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:19.919065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] PATH_INFO: `/v3/projects/d4513e8c02d44ea1a6b0baf406358e90` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:19.920161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4513e8c02d44ea1a6b0baf406358e90)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:19.922136 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:19.970787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4513e8c02d44ea1a6b0baf406358e90', 'request_id': 'req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.976284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4513e8c02d44ea1a6b0baf406358e90', 'request_id': 'req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.977886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:19.985300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4513e8c02d44ea1a6b0baf406358e90', 'request_id': 'req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.997039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] The token cache is being invalidate because project d4513e8c02d44ea1a6b0baf406358e90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:37:19.997322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.997955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-81552bb2-5f89-42d2-bcc1-53fa92d48ab4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:19.999440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1185/2370] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:37:19 2022] DELETE /identity/v3/projects/d4513e8c02d44ea1a6b0baf406358e90 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:20.004546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770429c9-0545-47d1-b7b5-a1f2227db50a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.005017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770429c9-0545-47d1-b7b5-a1f2227db50a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.005110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770429c9-0545-47d1-b7b5-a1f2227db50a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.026066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-770429c9-0545-47d1-b7b5-a1f2227db50a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:20.027330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-770429c9-0545-47d1-b7b5-a1f2227db50a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:20.090519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1186/2371] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:20.099369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c126084c-f04a-4025-a68e-287b661f737c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.099833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c126084c-f04a-4025-a68e-287b661f737c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.100235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c126084c-f04a-4025-a68e-287b661f737c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.120589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c126084c-f04a-4025-a68e-287b661f737c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:20.121794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c126084c-f04a-4025-a68e-287b661f737c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:20.130185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1186/2372] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:20.134602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770429c9-0545-47d1-b7b5-a1f2227db50a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:20.145007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770429c9-0545-47d1-b7b5-a1f2227db50a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:20.150814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6638c69-ef35-4e8f-945f-d4ee8758a63a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:20.151422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6638c69-ef35-4e8f-945f-d4ee8758a63a admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.151698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6638c69-ef35-4e8f-945f-d4ee8758a63a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.151965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6638c69-ef35-4e8f-945f-d4ee8758a63a admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.152610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6638c69-ef35-4e8f-945f-d4ee8758a63a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:37:20.152915 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6638c69-ef35-4e8f-945f-d4ee8758a63a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:20.154576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6638c69-ef35-4e8f-945f-d4ee8758a63a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:20.167330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1187/2373] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:37:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:37:20.171773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c126084c-f04a-4025-a68e-287b661f737c None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:20.181538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c126084c-f04a-4025-a68e-287b661f737c None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:20.187682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f188892-0008-4e5f-a619-530962ab93ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:20.188274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f188892-0008-4e5f-a619-530962ab93ea admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.188552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f188892-0008-4e5f-a619-530962ab93ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.188823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f188892-0008-4e5f-a619-530962ab93ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.189448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f188892-0008-4e5f-a619-530962ab93ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:20.191441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f188892-0008-4e5f-a619-530962ab93ea admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:20.227720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1187/2374] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:37:20 2022] POST /identity/v3/projects => generated 383 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:20.231972 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6638c69-ef35-4e8f-945f-d4ee8758a63a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:20.241937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6638c69-ef35-4e8f-945f-d4ee8758a63a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:20.273430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:20.309094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f367119e-1e04-4b7d-add3-8217acec7404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:20.309702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f367119e-1e04-4b7d-add3-8217acec7404 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.309975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f367119e-1e04-4b7d-add3-8217acec7404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.310274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f367119e-1e04-4b7d-add3-8217acec7404 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.310905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f367119e-1e04-4b7d-add3-8217acec7404 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:20.312835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f367119e-1e04-4b7d-add3-8217acec7404 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:20.337685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1188/2375] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:37:20 2022] POST /identity/v3/users => generated 292 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:20.342662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f188892-0008-4e5f-a619-530962ab93ea admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:20.357304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f188892-0008-4e5f-a619-530962ab93ea admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:20.365029 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4f973e-73ec-4075-8994-87a9d7c74128 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:20.365838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4f973e-73ec-4075-8994-87a9d7c74128 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.366153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4f973e-73ec-4075-8994-87a9d7c74128 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.366678 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4f973e-73ec-4075-8994-87a9d7c74128 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.367325 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4f973e-73ec-4075-8994-87a9d7c74128 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:20.368978 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4f973e-73ec-4075-8994-87a9d7c74128 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:20.379540 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1188/2376] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:20 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:20.384596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f367119e-1e04-4b7d-add3-8217acec7404 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:20.398382 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f367119e-1e04-4b7d-add3-8217acec7404 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:20.403511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969bacad-c789-49e9-a0c1-506b2cb06f01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:20.404104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969bacad-c789-49e9-a0c1-506b2cb06f01 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.404357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969bacad-c789-49e9-a0c1-506b2cb06f01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.404639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969bacad-c789-49e9-a0c1-506b2cb06f01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.405285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969bacad-c789-49e9-a0c1-506b2cb06f01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:20.406964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969bacad-c789-49e9-a0c1-506b2cb06f01 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:20.417181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1189/2377] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:20 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:20.421488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4f973e-73ec-4075-8994-87a9d7c74128 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:20.431967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4f973e-73ec-4075-8994-87a9d7c74128 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:20.437733 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9412c6-f4f0-4b46-8900-bec1cc289b2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:20.438401 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9412c6-f4f0-4b46-8900-bec1cc289b2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.438673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9412c6-f4f0-4b46-8900-bec1cc289b2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.438940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9412c6-f4f0-4b46-8900-bec1cc289b2e admin admin] PATH_INFO: `/v3/projects/4e0c6c9ccc4a4f4b83fb905335e72cf8/users/0c5dcdbdd9f3400e99a57524db80915d/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.455417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9412c6-f4f0-4b46-8900-bec1cc289b2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e0c6c9ccc4a4f4b83fb905335e72cf8, user_id=0c5dcdbdd9f3400e99a57524db80915d, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:20.458167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9412c6-f4f0-4b46-8900-bec1cc289b2e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:20.473292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1189/2378] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:20 2022] PUT /identity/v3/projects/4e0c6c9ccc4a4f4b83fb905335e72cf8/users/0c5dcdbdd9f3400e99a57524db80915d/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:20.484873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcdb180-eb66-4e10-8d0e-216a310721d9 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.485135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcdb180-eb66-4e10-8d0e-216a310721d9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.485407 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcdb180-eb66-4e10-8d0e-216a310721d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.514762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-1dcdb180-eb66-4e10-8d0e-216a310721d9 None None] MFA Rules not processed for user `0c5dcdbdd9f3400e99a57524db80915d`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:20.516045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1dcdb180-eb66-4e10-8d0e-216a310721d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:20.588657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1190/2379] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:20.593482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9412c6-f4f0-4b46-8900-bec1cc289b2e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:20.660792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9412c6-f4f0-4b46-8900-bec1cc289b2e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:20.666277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07956868-6feb-44f2-8270-e333fe7f0000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:20.666924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07956868-6feb-44f2-8270-e333fe7f0000 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.667200 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07956868-6feb-44f2-8270-e333fe7f0000 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.667470 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07956868-6feb-44f2-8270-e333fe7f0000 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.668083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07956868-6feb-44f2-8270-e333fe7f0000 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:20.670015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07956868-6feb-44f2-8270-e333fe7f0000 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:20.698828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1190/2380] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:37:20 2022] POST /identity/v3/projects => generated 383 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:20.703062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcdb180-eb66-4e10-8d0e-216a310721d9 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:20.715261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcdb180-eb66-4e10-8d0e-216a310721d9 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:20.775148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12406b3-98bd-47c6-92c2-8340d3785ae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:20.775681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12406b3-98bd-47c6-92c2-8340d3785ae0 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.775958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12406b3-98bd-47c6-92c2-8340d3785ae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.776221 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12406b3-98bd-47c6-92c2-8340d3785ae0 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.776850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12406b3-98bd-47c6-92c2-8340d3785ae0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:20.778796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12406b3-98bd-47c6-92c2-8340d3785ae0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:20.804267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1191/2381] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:37:20 2022] POST /identity/v3/users => generated 298 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:20.808612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07956868-6feb-44f2-8270-e333fe7f0000 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:20.819445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07956868-6feb-44f2-8270-e333fe7f0000 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:20.824272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408f0637-f007-4030-afc8-9542af1f57e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:20.824862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408f0637-f007-4030-afc8-9542af1f57e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.825147 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408f0637-f007-4030-afc8-9542af1f57e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.825414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408f0637-f007-4030-afc8-9542af1f57e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:20.826048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408f0637-f007-4030-afc8-9542af1f57e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:20.827679 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408f0637-f007-4030-afc8-9542af1f57e3 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:20.838689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1191/2382] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:37:20.842859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12406b3-98bd-47c6-92c2-8340d3785ae0 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:20.854092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12406b3-98bd-47c6-92c2-8340d3785ae0 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:20.858608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854c3e3d-5c2b-48ce-8566-0bbb986fbd4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:20.859252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854c3e3d-5c2b-48ce-8566-0bbb986fbd4a admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:20.859527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854c3e3d-5c2b-48ce-8566-0bbb986fbd4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:20.859796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854c3e3d-5c2b-48ce-8566-0bbb986fbd4a admin admin] PATH_INFO: `/v3/projects/a6249beae0e346f7a467ab631a26b3aa/users/0c15f04df8564779bac8dcac934a8f7d/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:21.002016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854c3e3d-5c2b-48ce-8566-0bbb986fbd4a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6249beae0e346f7a467ab631a26b3aa, user_id=0c15f04df8564779bac8dcac934a8f7d, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:21.004666 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854c3e3d-5c2b-48ce-8566-0bbb986fbd4a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:21.021758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1192/2383] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:20 2022] PUT /identity/v3/projects/a6249beae0e346f7a467ab631a26b3aa/users/0c15f04df8564779bac8dcac934a8f7d/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:21.026044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408f0637-f007-4030-afc8-9542af1f57e3 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:21.036105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408f0637-f007-4030-afc8-9542af1f57e3 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:21.092352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:21.099389 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0254cac-d197-4c5e-8992-4cdd1697463f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:21.099989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0254cac-d197-4c5e-8992-4cdd1697463f admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:21.100592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0254cac-d197-4c5e-8992-4cdd1697463f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:21.101142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0254cac-d197-4c5e-8992-4cdd1697463f admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:21.101431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0254cac-d197-4c5e-8992-4cdd1697463f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:21.103293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0254cac-d197-4c5e-8992-4cdd1697463f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:21.114476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1192/2384] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:21 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:21.120265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854c3e3d-5c2b-48ce-8566-0bbb986fbd4a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:21.136106 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854c3e3d-5c2b-48ce-8566-0bbb986fbd4a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:21.144669 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12de73e-f20d-4fb2-a422-5cd49303b240 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:21.145572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12de73e-f20d-4fb2-a422-5cd49303b240 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:21.145911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12de73e-f20d-4fb2-a422-5cd49303b240 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:21.146268 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12de73e-f20d-4fb2-a422-5cd49303b240 admin admin] PATH_INFO: `/v3/projects/a6249beae0e346f7a467ab631a26b3aa/users/0c15f04df8564779bac8dcac934a8f7d/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:21.148839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12de73e-f20d-4fb2-a422-5cd49303b240 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6249beae0e346f7a467ab631a26b3aa, user_id=0c15f04df8564779bac8dcac934a8f7d, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:21.151555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12de73e-f20d-4fb2-a422-5cd49303b240 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:21.153741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:21.164240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1193/2385] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:21 2022] PUT /identity/v3/projects/a6249beae0e346f7a467ab631a26b3aa/users/0c15f04df8564779bac8dcac934a8f7d/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:21.175003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9991810-29d1-470d-b3ba-07c48938b559 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:21.175368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9991810-29d1-470d-b3ba-07c48938b559 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:21.175757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9991810-29d1-470d-b3ba-07c48938b559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:21.204877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c9991810-29d1-470d-b3ba-07c48938b559 None None] MFA Rules not processed for user `0c15f04df8564779bac8dcac934a8f7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:21.206458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c9991810-29d1-470d-b3ba-07c48938b559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:21.277334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1193/2386] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:21 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:21.288783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12de73e-f20d-4fb2-a422-5cd49303b240 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:21.290787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e12de73e-f20d-4fb2-a422-5cd49303b240 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:21.359263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12de73e-f20d-4fb2-a422-5cd49303b240 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:21.365249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e0df0f-bb15-4cfc-8299-8a5f4aeda00e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:21.366539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e0df0f-bb15-4cfc-8299-8a5f4aeda00e service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:21.366894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e0df0f-bb15-4cfc-8299-8a5f4aeda00e service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:21.367194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e0df0f-bb15-4cfc-8299-8a5f4aeda00e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:21.382405 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e0df0f-bb15-4cfc-8299-8a5f4aeda00e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:21.383828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e0df0f-bb15-4cfc-8299-8a5f4aeda00e service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:21.393926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1194/2387] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:21 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:21.501476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9991810-29d1-470d-b3ba-07c48938b559 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:21.511493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9991810-29d1-470d-b3ba-07c48938b559 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:21.515482 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a68880-4d8f-45bf-a8df-458681082dac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:21.516086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a68880-4d8f-45bf-a8df-458681082dac service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:21.516361 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a68880-4d8f-45bf-a8df-458681082dac service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:21.516617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a68880-4d8f-45bf-a8df-458681082dac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:21.574792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a68880-4d8f-45bf-a8df-458681082dac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:21.576148 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a68880-4d8f-45bf-a8df-458681082dac service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:21.587111 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1194/2388] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:24.248518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e0df0f-bb15-4cfc-8299-8a5f4aeda00e service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:24.331537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e0df0f-bb15-4cfc-8299-8a5f4aeda00e service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:24.337275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:24.337857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:24.338147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:24.338425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] PATH_INFO: `/v3/users/0c5dcdbdd9f3400e99a57524db80915d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:24.339621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c5dcdbdd9f3400e99a57524db80915d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:24.341501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:24.404682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c5dcdbdd9f3400e99a57524db80915d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:24.414164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c5dcdbdd9f3400e99a57524db80915d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:24.414956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c5dcdbdd9f3400e99a57524db80915d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:24.415733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c5dcdbdd9f3400e99a57524db80915d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:24.422788 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c5dcdbdd9f3400e99a57524db80915d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:24.440249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1195/2389] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:37:24 2022] DELETE /identity/v3/users/0c5dcdbdd9f3400e99a57524db80915d => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:24.444891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa646ff-a1ba-467d-8711-7293fd0ccc47 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:24.445386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa646ff-a1ba-467d-8711-7293fd0ccc47 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:24.445386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa646ff-a1ba-467d-8711-7293fd0ccc47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:24.465090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8aa646ff-a1ba-467d-8711-7293fd0ccc47 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:24.466306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8aa646ff-a1ba-467d-8711-7293fd0ccc47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:24.528101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1195/2390] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:24.538087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:24.539650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:24.559395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:24.601349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d646c390-1bc6-433a-a159-ba882831803e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:24.605308 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f406756c-e569-43ca-934c-d5d03e447cd5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:24.605904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406756c-e569-43ca-934c-d5d03e447cd5 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:24.606198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406756c-e569-43ca-934c-d5d03e447cd5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:24.606453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f406756c-e569-43ca-934c-d5d03e447cd5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:24.616098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f406756c-e569-43ca-934c-d5d03e447cd5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:24.617500 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f406756c-e569-43ca-934c-d5d03e447cd5 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:24.630216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1196/2391] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:24.905756 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa646ff-a1ba-467d-8711-7293fd0ccc47 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:24.907416 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8aa646ff-a1ba-467d-8711-7293fd0ccc47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:24.919856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa646ff-a1ba-467d-8711-7293fd0ccc47 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:24.925162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:24.925877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:24.926402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:24.926523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] PATH_INFO: `/v3/projects/4e0c6c9ccc4a4f4b83fb905335e72cf8` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:24.927876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e0c6c9ccc4a4f4b83fb905335e72cf8)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:24.929697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:24.942029 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:24.988526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e0c6c9ccc4a4f4b83fb905335e72cf8', 'request_id': 'req-f3befa07-c998-4abd-8b15-62c0323817ae'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:24.994611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e0c6c9ccc4a4f4b83fb905335e72cf8', 'request_id': 'req-f3befa07-c998-4abd-8b15-62c0323817ae'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.003107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e0c6c9ccc4a4f4b83fb905335e72cf8', 'request_id': 'req-f3befa07-c998-4abd-8b15-62c0323817ae'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.015856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] The token cache is being invalidate because project 4e0c6c9ccc4a4f4b83fb905335e72cf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:37:25.016149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.016909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.018478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1196/2392] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:37:24 2022] DELETE /identity/v3/projects/4e0c6c9ccc4a4f4b83fb905335e72cf8 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:25.022835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f406756c-e569-43ca-934c-d5d03e447cd5 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:25.032676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f406756c-e569-43ca-934c-d5d03e447cd5 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:25.100821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:25.101460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:25.101739 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:25.102006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] PATH_INFO: `/v3/users/0c15f04df8564779bac8dcac934a8f7d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:25.103380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c15f04df8564779bac8dcac934a8f7d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:25.105379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:25.153614 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c15f04df8564779bac8dcac934a8f7d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.163223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c15f04df8564779bac8dcac934a8f7d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.163865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c15f04df8564779bac8dcac934a8f7d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.164518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c15f04df8564779bac8dcac934a8f7d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.173317 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-4d77f648-03cf-454b-bcdb-8a21e58becc0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c15f04df8564779bac8dcac934a8f7d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.197310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1197/2393] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:37:25 2022] DELETE /identity/v3/users/0c15f04df8564779bac8dcac934a8f7d => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:25.496882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:25.498417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:25.511616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3befa07-c998-4abd-8b15-62c0323817ae admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:25.577664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:25.578236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:25.578515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:25.578787 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] PATH_INFO: `/v3/projects/a6249beae0e346f7a467ab631a26b3aa` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:25.580275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6249beae0e346f7a467ab631a26b3aa)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:25.582218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:25.631646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6249beae0e346f7a467ab631a26b3aa', 'request_id': 'req-406b98c7-4ce8-4c54-95db-8a78d327f65c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.638590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6249beae0e346f7a467ab631a26b3aa', 'request_id': 'req-406b98c7-4ce8-4c54-95db-8a78d327f65c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.648310 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6249beae0e346f7a467ab631a26b3aa', 'request_id': 'req-406b98c7-4ce8-4c54-95db-8a78d327f65c'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.659956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] The token cache is being invalidate because project a6249beae0e346f7a467ab631a26b3aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:37:25.660239 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.660844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-406b98c7-4ce8-4c54-95db-8a78d327f65c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:25.662343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1197/2394] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:37:25 2022] DELETE /identity/v3/projects/a6249beae0e346f7a467ab631a26b3aa => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:25.667521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbd6664-e81b-41b3-92c9-c93a5a8b5fd3 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:25.668003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbd6664-e81b-41b3-92c9-c93a5a8b5fd3 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:25.668112 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbd6664-e81b-41b3-92c9-c93a5a8b5fd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:25.687686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-2dbd6664-e81b-41b3-92c9-c93a5a8b5fd3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:25.688860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2dbd6664-e81b-41b3-92c9-c93a5a8b5fd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:25.721096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:25.747682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1198/2395] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:25.755690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f8d92-0eba-46ba-97ba-d50527c90a06 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:25.755971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f8d92-0eba-46ba-97ba-d50527c90a06 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:25.756234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f8d92-0eba-46ba-97ba-d50527c90a06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:25.776718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-4e1f8d92-0eba-46ba-97ba-d50527c90a06 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:25.777928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4e1f8d92-0eba-46ba-97ba-d50527c90a06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:25.786237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1198/2396] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:25.790689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbd6664-e81b-41b3-92c9-c93a5a8b5fd3 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:25.800878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbd6664-e81b-41b3-92c9-c93a5a8b5fd3 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:25.806965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc9d03-b5b2-49e3-a88a-a3d852381fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:25.807586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc9d03-b5b2-49e3-a88a-a3d852381fcb admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:25.807859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc9d03-b5b2-49e3-a88a-a3d852381fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:25.808120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc9d03-b5b2-49e3-a88a-a3d852381fcb admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:25.808767 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc9d03-b5b2-49e3-a88a-a3d852381fcb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:37:25.809069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc9d03-b5b2-49e3-a88a-a3d852381fcb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:25.810733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc9d03-b5b2-49e3-a88a-a3d852381fcb admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:25.823037 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1199/2397] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:37:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:37:25.829403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1f8d92-0eba-46ba-97ba-d50527c90a06 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:25.841727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1f8d92-0eba-46ba-97ba-d50527c90a06 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:25.847035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4febdd-4bd2-44be-a5d8-39383711f90a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:25.847727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4febdd-4bd2-44be-a5d8-39383711f90a admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:25.848020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4febdd-4bd2-44be-a5d8-39383711f90a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:25.848286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4febdd-4bd2-44be-a5d8-39383711f90a admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:25.848909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4febdd-4bd2-44be-a5d8-39383711f90a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:25.850851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4febdd-4bd2-44be-a5d8-39383711f90a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:25.885274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1199/2398] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:37:25 2022] POST /identity/v3/projects => generated 381 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:25.889563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc9d03-b5b2-49e3-a88a-a3d852381fcb admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:25.900323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc9d03-b5b2-49e3-a88a-a3d852381fcb admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:25.963312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ea59f5-43ed-404f-acf3-1ae952dc2792 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:25.963961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ea59f5-43ed-404f-acf3-1ae952dc2792 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:25.964232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ea59f5-43ed-404f-acf3-1ae952dc2792 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:25.964494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ea59f5-43ed-404f-acf3-1ae952dc2792 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:25.965151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ea59f5-43ed-404f-acf3-1ae952dc2792 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:25.967199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ea59f5-43ed-404f-acf3-1ae952dc2792 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:25.994032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1200/2399] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:37:25 2022] POST /identity/v3/users => generated 291 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:25.998339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4febdd-4bd2-44be-a5d8-39383711f90a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:26.003923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:26.011152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4febdd-4bd2-44be-a5d8-39383711f90a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:26.017513 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1458a6af-c1a0-4393-9a60-d5cdab90580e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:26.018131 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1458a6af-c1a0-4393-9a60-d5cdab90580e admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.018435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1458a6af-c1a0-4393-9a60-d5cdab90580e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.018710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1458a6af-c1a0-4393-9a60-d5cdab90580e admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.019360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1458a6af-c1a0-4393-9a60-d5cdab90580e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:26.020982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1458a6af-c1a0-4393-9a60-d5cdab90580e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:26.031947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1200/2400] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:26.036082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ea59f5-43ed-404f-acf3-1ae952dc2792 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:26.046304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ea59f5-43ed-404f-acf3-1ae952dc2792 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:26.051611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf59fe88-19c3-43c1-9ad8-5833382c22e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:26.052214 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf59fe88-19c3-43c1-9ad8-5833382c22e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.052483 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf59fe88-19c3-43c1-9ad8-5833382c22e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.052774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf59fe88-19c3-43c1-9ad8-5833382c22e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.053409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf59fe88-19c3-43c1-9ad8-5833382c22e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:26.055029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf59fe88-19c3-43c1-9ad8-5833382c22e1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:26.065603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1201/2401] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:26.070076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1458a6af-c1a0-4393-9a60-d5cdab90580e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:26.081227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1458a6af-c1a0-4393-9a60-d5cdab90580e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:26.086715 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5893b25-97b0-453e-9812-1791c1c0d380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:26.087446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5893b25-97b0-453e-9812-1791c1c0d380 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.087727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5893b25-97b0-453e-9812-1791c1c0d380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.087999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5893b25-97b0-453e-9812-1791c1c0d380 admin admin] PATH_INFO: `/v3/projects/518089ebcb22470e8952baf2f5cf3cac/users/c901ce4b4556417ab65aec1ad1a59b5d/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.103152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5893b25-97b0-453e-9812-1791c1c0d380 admin admin] RBAC: Authorizing `identity:create_grant(project_id=518089ebcb22470e8952baf2f5cf3cac, user_id=c901ce4b4556417ab65aec1ad1a59b5d, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:26.105829 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5893b25-97b0-453e-9812-1791c1c0d380 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:26.121230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1201/2402] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:26 2022] PUT /identity/v3/projects/518089ebcb22470e8952baf2f5cf3cac/users/c901ce4b4556417ab65aec1ad1a59b5d/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:26.132454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d406d8c-6764-432c-96ed-57e7694f80ee None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.132747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d406d8c-6764-432c-96ed-57e7694f80ee None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.133011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d406d8c-6764-432c-96ed-57e7694f80ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.162960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0d406d8c-6764-432c-96ed-57e7694f80ee None None] MFA Rules not processed for user `c901ce4b4556417ab65aec1ad1a59b5d`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:26.164144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0d406d8c-6764-432c-96ed-57e7694f80ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:26.235985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1202/2403] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:26 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:26.242783 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5893b25-97b0-453e-9812-1791c1c0d380 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:26.311525 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5893b25-97b0-453e-9812-1791c1c0d380 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:26.317066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e389b2-2b0d-4011-af72-c81175ccc72b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:26.317668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e389b2-2b0d-4011-af72-c81175ccc72b admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.317942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e389b2-2b0d-4011-af72-c81175ccc72b admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.318237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e389b2-2b0d-4011-af72-c81175ccc72b admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.318879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e389b2-2b0d-4011-af72-c81175ccc72b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:26.320891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e389b2-2b0d-4011-af72-c81175ccc72b admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:26.366772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1202/2404] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:37:26 2022] POST /identity/v3/projects => generated 379 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:26.374179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d406d8c-6764-432c-96ed-57e7694f80ee None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:26.389482 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d406d8c-6764-432c-96ed-57e7694f80ee None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:26.420896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:26.450194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7b3342-fea5-4150-9090-630032b1d405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:26.450820 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7b3342-fea5-4150-9090-630032b1d405 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.451092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7b3342-fea5-4150-9090-630032b1d405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.451357 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7b3342-fea5-4150-9090-630032b1d405 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.452033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7b3342-fea5-4150-9090-630032b1d405 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:26.453935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7b3342-fea5-4150-9090-630032b1d405 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:26.478083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1203/2405] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:37:26 2022] POST /identity/v3/users => generated 296 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:26.482316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e389b2-2b0d-4011-af72-c81175ccc72b admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:26.492503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e389b2-2b0d-4011-af72-c81175ccc72b admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:26.497862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f5f1e7-73ce-497e-bdf9-527d67739900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:26.498513 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f5f1e7-73ce-497e-bdf9-527d67739900 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.498959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f5f1e7-73ce-497e-bdf9-527d67739900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.499283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f5f1e7-73ce-497e-bdf9-527d67739900 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.499958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f5f1e7-73ce-497e-bdf9-527d67739900 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:26.501728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f5f1e7-73ce-497e-bdf9-527d67739900 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:26.512773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1203/2406] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:26.517119 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7b3342-fea5-4150-9090-630032b1d405 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:26.527139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7b3342-fea5-4150-9090-630032b1d405 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:26.532956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f0fc2f-6eb3-4efd-aa15-50bc65280d8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:26.533603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f0fc2f-6eb3-4efd-aa15-50bc65280d8a admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.533870 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f0fc2f-6eb3-4efd-aa15-50bc65280d8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.534148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f0fc2f-6eb3-4efd-aa15-50bc65280d8a admin admin] PATH_INFO: `/v3/projects/d3bded18547843d786b88b82aa4f215e/users/82d9b090c5414fff9a98aa1ca28c38be/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.552381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f0fc2f-6eb3-4efd-aa15-50bc65280d8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3bded18547843d786b88b82aa4f215e, user_id=82d9b090c5414fff9a98aa1ca28c38be, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:26.556366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f0fc2f-6eb3-4efd-aa15-50bc65280d8a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:26.572637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1204/2407] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:26 2022] PUT /identity/v3/projects/d3bded18547843d786b88b82aa4f215e/users/82d9b090c5414fff9a98aa1ca28c38be/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:26.576813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f5f1e7-73ce-497e-bdf9-527d67739900 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:26.588039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f5f1e7-73ce-497e-bdf9-527d67739900 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:26.653442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd867aa-47d0-43a9-a732-da65d1906cde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:26.654083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd867aa-47d0-43a9-a732-da65d1906cde admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.654395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd867aa-47d0-43a9-a732-da65d1906cde admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.654662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd867aa-47d0-43a9-a732-da65d1906cde admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.655301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd867aa-47d0-43a9-a732-da65d1906cde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:26.656919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd867aa-47d0-43a9-a732-da65d1906cde admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:26.667337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1204/2408] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:26 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:26.672327 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f0fc2f-6eb3-4efd-aa15-50bc65280d8a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:26.683002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f0fc2f-6eb3-4efd-aa15-50bc65280d8a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:26.687798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a16db6-cfd7-4561-b83f-2ef0bf2723bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:26.688446 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a16db6-cfd7-4561-b83f-2ef0bf2723bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.688718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a16db6-cfd7-4561-b83f-2ef0bf2723bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.688986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a16db6-cfd7-4561-b83f-2ef0bf2723bb admin admin] PATH_INFO: `/v3/projects/d3bded18547843d786b88b82aa4f215e/users/82d9b090c5414fff9a98aa1ca28c38be/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.691149 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a16db6-cfd7-4561-b83f-2ef0bf2723bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3bded18547843d786b88b82aa4f215e, user_id=82d9b090c5414fff9a98aa1ca28c38be, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:26.693722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a16db6-cfd7-4561-b83f-2ef0bf2723bb admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:26.705082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1205/2409] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:26 2022] PUT /identity/v3/projects/d3bded18547843d786b88b82aa4f215e/users/82d9b090c5414fff9a98aa1ca28c38be/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:26.715511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770352c9-a2df-49ad-838d-b7de47fd65dc None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.715793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770352c9-a2df-49ad-838d-b7de47fd65dc None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.716067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770352c9-a2df-49ad-838d-b7de47fd65dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.744971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-770352c9-a2df-49ad-838d-b7de47fd65dc None None] MFA Rules not processed for user `82d9b090c5414fff9a98aa1ca28c38be`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:26.746173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-770352c9-a2df-49ad-838d-b7de47fd65dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:26.772693 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:26.817185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1205/2410] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:26 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:26.829103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a16db6-cfd7-4561-b83f-2ef0bf2723bb admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:26.831356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-56a16db6-cfd7-4561-b83f-2ef0bf2723bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:26.890440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a16db6-cfd7-4561-b83f-2ef0bf2723bb admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:26.896111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b20e59-78b1-4808-b253-c1bedf458553 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:26.896779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b20e59-78b1-4808-b253-c1bedf458553 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:26.897006 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b20e59-78b1-4808-b253-c1bedf458553 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:26.897275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b20e59-78b1-4808-b253-c1bedf458553 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:26.907091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b20e59-78b1-4808-b253-c1bedf458553 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:26.908543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b20e59-78b1-4808-b253-c1bedf458553 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:26.918419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1206/2411] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:26 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:26.994433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770352c9-a2df-49ad-838d-b7de47fd65dc None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:27.005087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770352c9-a2df-49ad-838d-b7de47fd65dc None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:27.009472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae8562f-a21c-471d-84f0-6adece4dc63f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:27.010127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae8562f-a21c-471d-84f0-6adece4dc63f service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:27.010414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae8562f-a21c-471d-84f0-6adece4dc63f service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:27.010684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae8562f-a21c-471d-84f0-6adece4dc63f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:27.075596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae8562f-a21c-471d-84f0-6adece4dc63f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:27.077040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae8562f-a21c-471d-84f0-6adece4dc63f service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:27.086712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1206/2412] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:26 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:27.362085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b20e59-78b1-4808-b253-c1bedf458553 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:27.424562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b20e59-78b1-4808-b253-c1bedf458553 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:27.429227 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:27.429860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:27.430347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:27.430617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] PATH_INFO: `/v3/users/c901ce4b4556417ab65aec1ad1a59b5d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:27.431762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c901ce4b4556417ab65aec1ad1a59b5d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:27.433668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:27.469984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:27.485007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c901ce4b4556417ab65aec1ad1a59b5d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:27.493342 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c901ce4b4556417ab65aec1ad1a59b5d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:27.493939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c901ce4b4556417ab65aec1ad1a59b5d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:27.494650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c901ce4b4556417ab65aec1ad1a59b5d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:27.501789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c901ce4b4556417ab65aec1ad1a59b5d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:27.520115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1207/2413] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:37:27 2022] DELETE /identity/v3/users/c901ce4b4556417ab65aec1ad1a59b5d => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:27.524824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527ce620-d593-4e04-8f37-d43c19cbc77d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:27.525327 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527ce620-d593-4e04-8f37-d43c19cbc77d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:27.525327 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527ce620-d593-4e04-8f37-d43c19cbc77d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:27.544042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-527ce620-d593-4e04-8f37-d43c19cbc77d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:27.545155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-527ce620-d593-4e04-8f37-d43c19cbc77d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:27.603938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1207/2414] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:27.613769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:27.615225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:27.672852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6180dc-d014-4167-8b8f-c39e81cb1ac5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:27.677047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53251fb3-5abc-4a18-a17c-6905acc794bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:27.677590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53251fb3-5abc-4a18-a17c-6905acc794bc service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:27.677859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53251fb3-5abc-4a18-a17c-6905acc794bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:27.678132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53251fb3-5abc-4a18-a17c-6905acc794bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:27.687845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53251fb3-5abc-4a18-a17c-6905acc794bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:27.689266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53251fb3-5abc-4a18-a17c-6905acc794bc service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:27.698591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1208/2415] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:27.981911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527ce620-d593-4e04-8f37-d43c19cbc77d None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:27.983675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-527ce620-d593-4e04-8f37-d43c19cbc77d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:28.002305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527ce620-d593-4e04-8f37-d43c19cbc77d None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:28.008731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:28.009521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:28.009806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:28.010095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] PATH_INFO: `/v3/projects/518089ebcb22470e8952baf2f5cf3cac` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:28.011282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=518089ebcb22470e8952baf2f5cf3cac)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:28.013208 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:28.062986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '518089ebcb22470e8952baf2f5cf3cac', 'request_id': 'req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.067552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:28.069006 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '518089ebcb22470e8952baf2f5cf3cac', 'request_id': 'req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.078930 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '518089ebcb22470e8952baf2f5cf3cac', 'request_id': 'req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.093163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] The token cache is being invalidate because project 518089ebcb22470e8952baf2f5cf3cac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:37:28.093450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.094172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.095718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1208/2416] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:37:27 2022] DELETE /identity/v3/projects/518089ebcb22470e8952baf2f5cf3cac => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:28.099929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53251fb3-5abc-4a18-a17c-6905acc794bc service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:28.110056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53251fb3-5abc-4a18-a17c-6905acc794bc service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:28.167312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:28.167939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:28.168217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:28.168487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] PATH_INFO: `/v3/users/82d9b090c5414fff9a98aa1ca28c38be` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:28.169657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82d9b090c5414fff9a98aa1ca28c38be)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:28.171569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:28.225575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82d9b090c5414fff9a98aa1ca28c38be'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.229023 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:28.234373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82d9b090c5414fff9a98aa1ca28c38be'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.235066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82d9b090c5414fff9a98aa1ca28c38be'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.236053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82d9b090c5414fff9a98aa1ca28c38be'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.244661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-634e4c33-b7eb-4b03-94e4-e0e4180dba63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82d9b090c5414fff9a98aa1ca28c38be'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.261839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1209/2417] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:37:28 2022] DELETE /identity/v3/users/82d9b090c5414fff9a98aa1ca28c38be => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:28.566184 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:28.567806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:28.582708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dc8642-b26b-49ca-a1c7-6ee43aec43dc admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:28.644378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:28.645115 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:28.645426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:28.645695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] PATH_INFO: `/v3/projects/d3bded18547843d786b88b82aa4f215e` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:28.646827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3bded18547843d786b88b82aa4f215e)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:28.648776 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:28.700989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3bded18547843d786b88b82aa4f215e', 'request_id': 'req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.707286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3bded18547843d786b88b82aa4f215e', 'request_id': 'req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.717484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3bded18547843d786b88b82aa4f215e', 'request_id': 'req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.730229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] The token cache is being invalidate because project d3bded18547843d786b88b82aa4f215e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:37:28.730487 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.731153 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9698fe92-0a9f-44b0-b40b-5d32b9b5cffb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:37:28.733003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1209/2418] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:37:28 2022] DELETE /identity/v3/projects/d3bded18547843d786b88b82aa4f215e => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:28.738378 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b9000b-b842-4e8b-8655-cbde18aa6864 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:28.738899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b9000b-b842-4e8b-8655-cbde18aa6864 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:28.738995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b9000b-b842-4e8b-8655-cbde18aa6864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:28.761837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-61b9000b-b842-4e8b-8655-cbde18aa6864 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:28.763254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-61b9000b-b842-4e8b-8655-cbde18aa6864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:28.827700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1210/2419] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:28.836640 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc7d5d4-d03f-4aae-b228-a56417135a8a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:28.836960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc7d5d4-d03f-4aae-b228-a56417135a8a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:28.837237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc7d5d4-d03f-4aae-b228-a56417135a8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:28.859575 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2bc7d5d4-d03f-4aae-b228-a56417135a8a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:28.860805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2bc7d5d4-d03f-4aae-b228-a56417135a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:28.870497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1210/2420] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:28.874907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b9000b-b842-4e8b-8655-cbde18aa6864 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:28.893095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b9000b-b842-4e8b-8655-cbde18aa6864 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:28.899650 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6983ab-f5da-408d-834b-2b41aed869a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:28.900272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6983ab-f5da-408d-834b-2b41aed869a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:28.900550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6983ab-f5da-408d-834b-2b41aed869a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:28.900819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6983ab-f5da-408d-834b-2b41aed869a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:28.901473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6983ab-f5da-408d-834b-2b41aed869a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:37:28.901776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6983ab-f5da-408d-834b-2b41aed869a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:28.903423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6983ab-f5da-408d-834b-2b41aed869a2 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:28.922075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1211/2421] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:37:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:37:28.926793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc7d5d4-d03f-4aae-b228-a56417135a8a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:28.937454 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc7d5d4-d03f-4aae-b228-a56417135a8a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:28.943834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1520ac8f-9cd4-4f94-8ed5-a0371d3cf175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:28.944433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1520ac8f-9cd4-4f94-8ed5-a0371d3cf175 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:28.944705 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1520ac8f-9cd4-4f94-8ed5-a0371d3cf175 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:28.944972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1520ac8f-9cd4-4f94-8ed5-a0371d3cf175 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:28.945767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1520ac8f-9cd4-4f94-8ed5-a0371d3cf175 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:28.948119 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1520ac8f-9cd4-4f94-8ed5-a0371d3cf175 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:28.985268 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1211/2422] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:37:28 2022] POST /identity/v3/projects => generated 371 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:28.990895 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6983ab-f5da-408d-834b-2b41aed869a2 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:29.001428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6983ab-f5da-408d-834b-2b41aed869a2 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:29.060662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f777760-2e0a-428a-a6c4-6a7a2185955a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:29.061270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f777760-2e0a-428a-a6c4-6a7a2185955a admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:29.061540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f777760-2e0a-428a-a6c4-6a7a2185955a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:29.061807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f777760-2e0a-428a-a6c4-6a7a2185955a admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:29.062461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f777760-2e0a-428a-a6c4-6a7a2185955a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:29.064355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f777760-2e0a-428a-a6c4-6a7a2185955a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:29.088578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1212/2423] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:37:28 2022] POST /identity/v3/users => generated 286 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:37:29.093012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1520ac8f-9cd4-4f94-8ed5-a0371d3cf175 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:29.104446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1520ac8f-9cd4-4f94-8ed5-a0371d3cf175 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:29.111766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d895fb4c-68e0-4c58-af0d-2398d74cd19a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:29.112395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d895fb4c-68e0-4c58-af0d-2398d74cd19a admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:29.112669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d895fb4c-68e0-4c58-af0d-2398d74cd19a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:29.112933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d895fb4c-68e0-4c58-af0d-2398d74cd19a admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:29.113563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d895fb4c-68e0-4c58-af0d-2398d74cd19a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:29.115244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d895fb4c-68e0-4c58-af0d-2398d74cd19a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:29.126265 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1212/2424] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:29 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:29.130394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f777760-2e0a-428a-a6c4-6a7a2185955a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:29.141053 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f777760-2e0a-428a-a6c4-6a7a2185955a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:29.146307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4642ce6-4168-471b-a7e0-6e3225ad519a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:29.146906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4642ce6-4168-471b-a7e0-6e3225ad519a admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:29.147198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4642ce6-4168-471b-a7e0-6e3225ad519a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:29.147460 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4642ce6-4168-471b-a7e0-6e3225ad519a admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:29.148095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4642ce6-4168-471b-a7e0-6e3225ad519a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:29.149810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4642ce6-4168-471b-a7e0-6e3225ad519a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:29.160689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1213/2425] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:37:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:37:29.165079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d895fb4c-68e0-4c58-af0d-2398d74cd19a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:29.175684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d895fb4c-68e0-4c58-af0d-2398d74cd19a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:29.180753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b28679-6f91-4252-ac2c-c9d3c4138ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:29.181399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b28679-6f91-4252-ac2c-c9d3c4138ba4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:29.181677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b28679-6f91-4252-ac2c-c9d3c4138ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:29.181978 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b28679-6f91-4252-ac2c-c9d3c4138ba4 admin admin] PATH_INFO: `/v3/projects/249dabd49af142558f6d5e0b8a72959d/users/ee3f3213553f4d06ad23ad3f107133b9/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:29.198255 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b28679-6f91-4252-ac2c-c9d3c4138ba4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=249dabd49af142558f6d5e0b8a72959d, user_id=ee3f3213553f4d06ad23ad3f107133b9, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:29.200949 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b28679-6f91-4252-ac2c-c9d3c4138ba4 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:29.216698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1213/2426] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:37:29 2022] PUT /identity/v3/projects/249dabd49af142558f6d5e0b8a72959d/users/ee3f3213553f4d06ad23ad3f107133b9/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:37:29.286568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc317b2-7344-49f8-8ab4-9700fcd9bbda None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:29.286816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc317b2-7344-49f8-8ab4-9700fcd9bbda None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:29.287025 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc317b2-7344-49f8-8ab4-9700fcd9bbda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:29.316381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-5cc317b2-7344-49f8-8ab4-9700fcd9bbda None None] MFA Rules not processed for user `ee3f3213553f4d06ad23ad3f107133b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:37:29.317696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5cc317b2-7344-49f8-8ab4-9700fcd9bbda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:29.342855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:29.385651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1214/2427] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:37:29 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:37:29.397657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b28679-6f91-4252-ac2c-c9d3c4138ba4 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:37:29.399976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-28b28679-6f91-4252-ac2c-c9d3c4138ba4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:37:29.414458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:29.462190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b28679-6f91-4252-ac2c-c9d3c4138ba4 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:37:29.466340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9deefcf2-6d2f-4036-adc3-9b8241067b08 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce646dcb8124685ac7a1c1eca664d04', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:37:29.466948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9deefcf2-6d2f-4036-adc3-9b8241067b08 service glance] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:37:29.467218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9deefcf2-6d2f-4036-adc3-9b8241067b08 service glance] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:37:29.467484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9deefcf2-6d2f-4036-adc3-9b8241067b08 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:37:29.476040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9deefcf2-6d2f-4036-adc3-9b8241067b08 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:37:29.477562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9deefcf2-6d2f-4036-adc3-9b8241067b08 service glance] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:37:29.486262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1214/2428] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:37:29 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:37:39.382801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=16 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:37:39.482329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=21 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:03.161016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f6892-6786-4662-b09e-2e0a5316c31d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:03.161546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f6892-6786-4662-b09e-2e0a5316c31d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:03.161546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614f6892-6786-4662-b09e-2e0a5316c31d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:03.201836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-614f6892-6786-4662-b09e-2e0a5316c31d None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:03.203415 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-614f6892-6786-4662-b09e-2e0a5316c31d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:03.286889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1215/2429] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:38:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:38:03.344897 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9deefcf2-6d2f-4036-adc3-9b8241067b08 service glance] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:03.346822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9deefcf2-6d2f-4036-adc3-9b8241067b08 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:03.448937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9deefcf2-6d2f-4036-adc3-9b8241067b08 service glance] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:03.455576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db8263a-eedc-4c49-948a-610c6e023df4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:03.456185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db8263a-eedc-4c49-948a-610c6e023df4 service swift] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:03.456463 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db8263a-eedc-4c49-948a-610c6e023df4 service swift] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:03.456732 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db8263a-eedc-4c49-948a-610c6e023df4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:03.465502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db8263a-eedc-4c49-948a-610c6e023df4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:03.467149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db8263a-eedc-4c49-948a-610c6e023df4 service swift] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:03.475792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1215/2430] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:38:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:38:05.960270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614f6892-6786-4662-b09e-2e0a5316c31d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:06.055760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614f6892-6786-4662-b09e-2e0a5316c31d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:06.061002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:06.061640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:06.061910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:06.062204 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] PATH_INFO: `/v3/users/ee3f3213553f4d06ad23ad3f107133b9` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:06.063333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee3f3213553f4d06ad23ad3f107133b9)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:06.065477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:06.143861 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:06.145568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee3f3213553f4d06ad23ad3f107133b9'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:06.156836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee3f3213553f4d06ad23ad3f107133b9'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:06.157656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee3f3213553f4d06ad23ad3f107133b9'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:06.158303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee3f3213553f4d06ad23ad3f107133b9'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:06.165988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee3f3213553f4d06ad23ad3f107133b9'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:06.184473 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1216/2431] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:38:05 2022] DELETE /identity/v3/users/ee3f3213553f4d06ad23ad3f107133b9 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:38:06.189108 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b8c3d8-cb30-43eb-bd5e-52acf747d617 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:06.189671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b8c3d8-cb30-43eb-bd5e-52acf747d617 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:06.189671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b8c3d8-cb30-43eb-bd5e-52acf747d617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:06.236633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-58b8c3d8-cb30-43eb-bd5e-52acf747d617 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:06.238034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-58b8c3d8-cb30-43eb-bd5e-52acf747d617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:06.306615 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1216/2432] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:38:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:06.316500 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:06.318035 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:06.387469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1baa8c8-cf50-452d-a9cc-ccffbd9b2176 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:06.391668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80edba8a-11d4-4271-8e27-7866fbdd6b4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:06.392447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80edba8a-11d4-4271-8e27-7866fbdd6b4c service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:06.392771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80edba8a-11d4-4271-8e27-7866fbdd6b4c service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:06.393128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80edba8a-11d4-4271-8e27-7866fbdd6b4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:06.403760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80edba8a-11d4-4271-8e27-7866fbdd6b4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:06.405504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80edba8a-11d4-4271-8e27-7866fbdd6b4c service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:06.424790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1217/2433] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:38:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:38:06.846841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b8c3d8-cb30-43eb-bd5e-52acf747d617 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:06.848280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-58b8c3d8-cb30-43eb-bd5e-52acf747d617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:06.862881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b8c3d8-cb30-43eb-bd5e-52acf747d617 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:06.868271 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:06.868891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:06.869174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:06.869446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] PATH_INFO: `/v3/projects/249dabd49af142558f6d5e0b8a72959d` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:06.878240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=249dabd49af142558f6d5e0b8a72959d)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:06.880347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:06.929183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:06.939124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '249dabd49af142558f6d5e0b8a72959d', 'request_id': 'req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:06.947833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '249dabd49af142558f6d5e0b8a72959d', 'request_id': 'req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:06.958487 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '249dabd49af142558f6d5e0b8a72959d', 'request_id': 'req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:06.972441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] The token cache is being invalidate because project 249dabd49af142558f6d5e0b8a72959d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:38:06.972724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:06.973415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-9eef8b60-bfa0-4a4a-9ba9-3ee25872e7f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:06.978623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1217/2434] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:38:06 2022] DELETE /identity/v3/projects/249dabd49af142558f6d5e0b8a72959d => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:38:06.980270 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7c5df-913c-400b-bb79-70113e9b3dd0 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:06.981071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7c5df-913c-400b-bb79-70113e9b3dd0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:06.981340 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7c5df-913c-400b-bb79-70113e9b3dd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:07.037260 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c1a7c5df-913c-400b-bb79-70113e9b3dd0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:07.038522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c1a7c5df-913c-400b-bb79-70113e9b3dd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:07.111741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1218/2435] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:38:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:07.113366 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2c0ea4-3a05-4e62-88f3-c3e5b2f0a958 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:07.113646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2c0ea4-3a05-4e62-88f3-c3e5b2f0a958 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:07.113914 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2c0ea4-3a05-4e62-88f3-c3e5b2f0a958 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:07.137514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-9a2c0ea4-3a05-4e62-88f3-c3e5b2f0a958 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:07.138824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9a2c0ea4-3a05-4e62-88f3-c3e5b2f0a958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:07.147648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1218/2436] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:38:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:07.154391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a7c5df-913c-400b-bb79-70113e9b3dd0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:07.166243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a7c5df-913c-400b-bb79-70113e9b3dd0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:07.175786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed669123-9f7e-4431-9712-b714a25628b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:07.176461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed669123-9f7e-4431-9712-b714a25628b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:07.176736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed669123-9f7e-4431-9712-b714a25628b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:07.177002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed669123-9f7e-4431-9712-b714a25628b3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:07.177663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed669123-9f7e-4431-9712-b714a25628b3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:38:07.177965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed669123-9f7e-4431-9712-b714a25628b3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:07.179611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed669123-9f7e-4431-9712-b714a25628b3 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:07.192863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1219/2437] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:38:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:38:07.202963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2c0ea4-3a05-4e62-88f3-c3e5b2f0a958 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:07.214017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2c0ea4-3a05-4e62-88f3-c3e5b2f0a958 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:07.220742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dc4f22-580a-4f23-aed9-6f43883aa947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:07.221810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dc4f22-580a-4f23-aed9-6f43883aa947 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:07.222288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dc4f22-580a-4f23-aed9-6f43883aa947 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:07.222728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dc4f22-580a-4f23-aed9-6f43883aa947 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:07.223550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dc4f22-580a-4f23-aed9-6f43883aa947 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:07.226476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dc4f22-580a-4f23-aed9-6f43883aa947 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:07.261890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1219/2438] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:38:07 2022] POST /identity/v3/projects => generated 363 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:38:07.268265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed669123-9f7e-4431-9712-b714a25628b3 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:07.281477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed669123-9f7e-4431-9712-b714a25628b3 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:07.321531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:07.365119 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b6b316-27cb-40f1-b1b8-592f50ec3dc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:07.365654 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b6b316-27cb-40f1-b1b8-592f50ec3dc0 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:07.365937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b6b316-27cb-40f1-b1b8-592f50ec3dc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:07.366245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b6b316-27cb-40f1-b1b8-592f50ec3dc0 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:07.367096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b6b316-27cb-40f1-b1b8-592f50ec3dc0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:07.369013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b6b316-27cb-40f1-b1b8-592f50ec3dc0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:07.397054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1220/2439] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:38:07 2022] POST /identity/v3/users => generated 282 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:38:07.401547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dc4f22-580a-4f23-aed9-6f43883aa947 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:07.413013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dc4f22-580a-4f23-aed9-6f43883aa947 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:07.418097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6be6ae-edb7-490a-99b4-1aec51b8ed18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:07.418754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6be6ae-edb7-490a-99b4-1aec51b8ed18 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:07.419034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6be6ae-edb7-490a-99b4-1aec51b8ed18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:07.419329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6be6ae-edb7-490a-99b4-1aec51b8ed18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:07.419950 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6be6ae-edb7-490a-99b4-1aec51b8ed18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:07.421574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6be6ae-edb7-490a-99b4-1aec51b8ed18 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:07.435740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1220/2440] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:38:07 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:38:07.437039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b6b316-27cb-40f1-b1b8-592f50ec3dc0 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:07.451527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b6b316-27cb-40f1-b1b8-592f50ec3dc0 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:07.457965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0166ea0-1101-4648-94e8-c458b4462f14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:07.458902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0166ea0-1101-4648-94e8-c458b4462f14 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:07.459189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0166ea0-1101-4648-94e8-c458b4462f14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:07.460517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0166ea0-1101-4648-94e8-c458b4462f14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:07.460517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0166ea0-1101-4648-94e8-c458b4462f14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:07.462156 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0166ea0-1101-4648-94e8-c458b4462f14 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:07.475891 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1221/2441] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:38:07 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:38:07.480187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6be6ae-edb7-490a-99b4-1aec51b8ed18 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:07.491922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6be6ae-edb7-490a-99b4-1aec51b8ed18 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:07.506969 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3871ebc9-5f7f-4005-954b-56012c25b04b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:07.507671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871ebc9-5f7f-4005-954b-56012c25b04b admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:07.507951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871ebc9-5f7f-4005-954b-56012c25b04b admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:07.508226 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871ebc9-5f7f-4005-954b-56012c25b04b admin admin] PATH_INFO: `/v3/projects/4edd6fb784c14fd584ebe852a053526e/users/634a6901fcec4d7da0cd3ba830f3fb7b/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:07.526826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3871ebc9-5f7f-4005-954b-56012c25b04b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4edd6fb784c14fd584ebe852a053526e, user_id=634a6901fcec4d7da0cd3ba830f3fb7b, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:07.530580 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3871ebc9-5f7f-4005-954b-56012c25b04b admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:07.553755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1221/2442] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:38:07 2022] PUT /identity/v3/projects/4edd6fb784c14fd584ebe852a053526e/users/634a6901fcec4d7da0cd3ba830f3fb7b/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:38:07.566594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fdfe3f-5b67-4cd1-9131-93b1e8aa7061 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:07.566967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fdfe3f-5b67-4cd1-9131-93b1e8aa7061 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:07.567313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fdfe3f-5b67-4cd1-9131-93b1e8aa7061 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:07.604752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-25fdfe3f-5b67-4cd1-9131-93b1e8aa7061 None None] MFA Rules not processed for user `634a6901fcec4d7da0cd3ba830f3fb7b`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:07.606324 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-25fdfe3f-5b67-4cd1-9131-93b1e8aa7061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:07.696842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1222/2443] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:38:07 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:07.707130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3871ebc9-5f7f-4005-954b-56012c25b04b admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:07.708979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3871ebc9-5f7f-4005-954b-56012c25b04b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:07.790277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3871ebc9-5f7f-4005-954b-56012c25b04b admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:07.795102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5acd261-8258-4df5-baf7-1b3c36b7e484 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:07.796456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5acd261-8258-4df5-baf7-1b3c36b7e484 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:07.796953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5acd261-8258-4df5-baf7-1b3c36b7e484 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:07.797503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5acd261-8258-4df5-baf7-1b3c36b7e484 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:07.812123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5acd261-8258-4df5-baf7-1b3c36b7e484 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:07.814409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5acd261-8258-4df5-baf7-1b3c36b7e484 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:07.850325 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1222/2444] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:38:07 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:38:17.688568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=53,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:17.825046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=64,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:19.243728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fdfe3f-5b67-4cd1-9131-93b1e8aa7061 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:19.245349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-25fdfe3f-5b67-4cd1-9131-93b1e8aa7061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:19.349372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fdfe3f-5b67-4cd1-9131-93b1e8aa7061 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:19.358811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775d6af1-b61b-47eb-a4b8-8ffabb79ef1c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:19.359694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775d6af1-b61b-47eb-a4b8-8ffabb79ef1c service placement] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:19.360047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775d6af1-b61b-47eb-a4b8-8ffabb79ef1c service placement] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:19.360419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775d6af1-b61b-47eb-a4b8-8ffabb79ef1c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:19.363681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-775d6af1-b61b-47eb-a4b8-8ffabb79ef1c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:19.375495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775d6af1-b61b-47eb-a4b8-8ffabb79ef1c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:19.377076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775d6af1-b61b-47eb-a4b8-8ffabb79ef1c service placement] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:19.388912 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1223/2445] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:38:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:38:20.206584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5acd261-8258-4df5-baf7-1b3c36b7e484 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:20.208126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e5acd261-8258-4df5-baf7-1b3c36b7e484 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:20.308235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5acd261-8258-4df5-baf7-1b3c36b7e484 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:20.313714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc1d4cd-eaa1-4723-a337-dd7279f352ad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:20.314452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc1d4cd-eaa1-4723-a337-dd7279f352ad service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:20.314729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc1d4cd-eaa1-4723-a337-dd7279f352ad service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:20.314998 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc1d4cd-eaa1-4723-a337-dd7279f352ad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:20.316710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6cc1d4cd-eaa1-4723-a337-dd7279f352ad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:20.376591 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc1d4cd-eaa1-4723-a337-dd7279f352ad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:20.378204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc1d4cd-eaa1-4723-a337-dd7279f352ad service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:20.388138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1223/2446] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:38:20 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:38:20.957969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed915f4-2815-41e3-a808-8ded882cb737 None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:20.958521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed915f4-2815-41e3-a808-8ded882cb737 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:20.958982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed915f4-2815-41e3-a808-8ded882cb737 None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:20.960837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1224/2447] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:38:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:38:20.967695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b3cbb0-5022-46d1-8545-eb8de32333e9 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:20.967972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b3cbb0-5022-46d1-8545-eb8de32333e9 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:20.968237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b3cbb0-5022-46d1-8545-eb8de32333e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:21.008400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d5b3cbb0-5022-46d1-8545-eb8de32333e9 None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:21.009735 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d5b3cbb0-5022-46d1-8545-eb8de32333e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:21.026050 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1224/2448] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:38:20 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:21.034354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed915f4-2815-41e3-a808-8ded882cb737 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:21.040122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0ed915f4-2815-41e3-a808-8ded882cb737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:21.123641 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed915f4-2815-41e3-a808-8ded882cb737 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:21.127226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c3f8e3-dd44-446b-8294-5f1922eea0dc service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce646dcb8124685ac7a1c1eca664d04', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:21.127929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3f8e3-dd44-446b-8294-5f1922eea0dc service glance] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:21.128198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3f8e3-dd44-446b-8294-5f1922eea0dc service glance] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:21.128462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3f8e3-dd44-446b-8294-5f1922eea0dc service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:21.138750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c3f8e3-dd44-446b-8294-5f1922eea0dc service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:21.140100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c3f8e3-dd44-446b-8294-5f1922eea0dc service glance] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:21.150636 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1225/2449] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:38:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:38:21.222646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0b220d-b8b9-4edf-9609-305c5d4ee585 None None] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:21.223218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0b220d-b8b9-4edf-9609-305c5d4ee585 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:21.223535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0b220d-b8b9-4edf-9609-305c5d4ee585 None None] PATH_INFO: `/` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:21.225035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1225/2450] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:38:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:38:21.230422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e589bae-d2d6-492b-816e-1d3f6f160ac3 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:21.231150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e589bae-d2d6-492b-816e-1d3f6f160ac3 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:21.231276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e589bae-d2d6-492b-816e-1d3f6f160ac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:21.263375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-9e589bae-d2d6-492b-816e-1d3f6f160ac3 None None] MFA Rules not processed for user `171e916244804add8f18b44b77b03d72`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:21.264627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9e589bae-d2d6-492b-816e-1d3f6f160ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:21.325724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1226/2451] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:38:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:21.338288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0b220d-b8b9-4edf-9609-305c5d4ee585 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:21.339991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ce0b220d-b8b9-4edf-9609-305c5d4ee585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:21.354536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0b220d-b8b9-4edf-9609-305c5d4ee585 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:21.358285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f816e84-9a3c-4c94-989c-31df5f4426d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:21.358892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f816e84-9a3c-4c94-989c-31df5f4426d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:21.359171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f816e84-9a3c-4c94-989c-31df5f4426d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:21.359455 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f816e84-9a3c-4c94-989c-31df5f4426d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:21.367642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f816e84-9a3c-4c94-989c-31df5f4426d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:21.369421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f816e84-9a3c-4c94-989c-31df5f4426d2 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:21.380337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1226/2452] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:38:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:38:21.467984 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5da685-b39b-4919-9c94-ee28bd73687c None None] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:21.468282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5da685-b39b-4919-9c94-ee28bd73687c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:21.468618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5da685-b39b-4919-9c94-ee28bd73687c None None] PATH_INFO: `/` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:21.469976 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1227/2453] 10.222.0.16 () {54 vars in 921 bytes} [Tue Sep 13 15:38:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 13 15:38:21.476096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cdac0e-9a0d-49d0-8527-56334bfc0483 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:21.476096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cdac0e-9a0d-49d0-8527-56334bfc0483 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:21.476302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cdac0e-9a0d-49d0-8527-56334bfc0483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:21.500114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c5cdac0e-9a0d-49d0-8527-56334bfc0483 None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:21.501305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c5cdac0e-9a0d-49d0-8527-56334bfc0483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:21.511498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1227/2454] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Sep 13 15:38:21 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:21.522571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5da685-b39b-4919-9c94-ee28bd73687c None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:21.524591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-df5da685-b39b-4919-9c94-ee28bd73687c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:21.532933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:21.539747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5da685-b39b-4919-9c94-ee28bd73687c None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:21.543440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8ad0df-1ed2-4bc8-ba16-24e916bc38d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:21.544036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8ad0df-1ed2-4bc8-ba16-24e916bc38d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:21.544307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8ad0df-1ed2-4bc8-ba16-24e916bc38d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:21.544572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8ad0df-1ed2-4bc8-ba16-24e916bc38d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:21.554844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8ad0df-1ed2-4bc8-ba16-24e916bc38d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:21.556251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8ad0df-1ed2-4bc8-ba16-24e916bc38d6 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:21.566851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1228/2455] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:38:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:38:23.385681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6204ed0-4213-4fac-aaf1-b110181e7a4b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:23.386555 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6204ed0-4213-4fac-aaf1-b110181e7a4b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:23.387008 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6204ed0-4213-4fac-aaf1-b110181e7a4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:23.407718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c6204ed0-4213-4fac-aaf1-b110181e7a4b None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:23.409309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c6204ed0-4213-4fac-aaf1-b110181e7a4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:23.466809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:23.486048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1228/2456] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:38:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:23.499505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8ad0df-1ed2-4bc8-ba16-24e916bc38d6 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:23.501007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7a8ad0df-1ed2-4bc8-ba16-24e916bc38d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:23.574372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8ad0df-1ed2-4bc8-ba16-24e916bc38d6 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:23.581180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdc4f4a-638d-43c8-a194-7e05b0702a18 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:23.581454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdc4f4a-638d-43c8-a194-7e05b0702a18 service swift] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:23.581761 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdc4f4a-638d-43c8-a194-7e05b0702a18 service swift] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:23.582094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdc4f4a-638d-43c8-a194-7e05b0702a18 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:23.591032 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdc4f4a-638d-43c8-a194-7e05b0702a18 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:23.592606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdc4f4a-638d-43c8-a194-7e05b0702a18 service swift] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:23.600831 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1229/2457] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:38:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:38:33.478557 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:33.597536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=37 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:45.130919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a52c5-e8eb-4bf5-8a6e-59407db46727 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:45.131581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a52c5-e8eb-4bf5-8a6e-59407db46727 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:45.131581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a52c5-e8eb-4bf5-8a6e-59407db46727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:45.154296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3d5a52c5-e8eb-4bf5-8a6e-59407db46727 None None] MFA Rules not processed for user `eac26eaa3b1a4c8c849a45ed9ba20301`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:45.155509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3d5a52c5-e8eb-4bf5-8a6e-59407db46727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:45.164087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1229/2458] 10.222.0.16 () {58 vars in 967 bytes} [Tue Sep 13 15:38:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:45.178185 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdc4f4a-638d-43c8-a194-7e05b0702a18 service swift] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:45.188583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdc4f4a-638d-43c8-a194-7e05b0702a18 service swift] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:45.194458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8017c885-0928-4c83-851b-8f2116126f42 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35ece665f492449282ab03ad91f0b254', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:45.195063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8017c885-0928-4c83-851b-8f2116126f42 service swift] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:45.195334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8017c885-0928-4c83-851b-8f2116126f42 service swift] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:45.195600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8017c885-0928-4c83-851b-8f2116126f42 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:45.209007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8017c885-0928-4c83-851b-8f2116126f42 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:45.210718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8017c885-0928-4c83-851b-8f2116126f42 service swift] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:45.219258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1230/2459] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Sep 13 15:38:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 13 15:38:50.506522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75ccec9-a72e-4604-9195-8440673c00e0 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:50.507096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75ccec9-a72e-4604-9195-8440673c00e0 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:50.507096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75ccec9-a72e-4604-9195-8440673c00e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:50.527063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e75ccec9-a72e-4604-9195-8440673c00e0 None None] MFA Rules not processed for user `f48ce3ac32e741ef98e5025fd23bbaa0`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:50.528261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e75ccec9-a72e-4604-9195-8440673c00e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:50.539513 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1230/2460] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:38:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:53.560735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8017c885-0928-4c83-851b-8f2116126f42 service swift] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:53.641411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8017c885-0928-4c83-851b-8f2116126f42 service swift] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:53.647371 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:53.648470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:53.648470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:53.648665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] PATH_INFO: `/v3/users/634a6901fcec4d7da0cd3ba830f3fb7b` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:53.649910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] RBAC: Authorizing `identity:delete_user(user_id=634a6901fcec4d7da0cd3ba830f3fb7b)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:53.651985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:53.746855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '634a6901fcec4d7da0cd3ba830f3fb7b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:53.760996 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '634a6901fcec4d7da0cd3ba830f3fb7b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:53.761759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '634a6901fcec4d7da0cd3ba830f3fb7b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:53.762394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '634a6901fcec4d7da0cd3ba830f3fb7b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:53.788142 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '634a6901fcec4d7da0cd3ba830f3fb7b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:53.823798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1231/2461] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:38:53 2022] DELETE /identity/v3/users/634a6901fcec4d7da0cd3ba830f3fb7b => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:38:53.828466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91ea7e0-703e-4729-a99e-b5c701eb7c00 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:53.828922 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91ea7e0-703e-4729-a99e-b5c701eb7c00 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:53.829021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91ea7e0-703e-4729-a99e-b5c701eb7c00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:53.859132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d91ea7e0-703e-4729-a99e-b5c701eb7c00 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:53.860428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d91ea7e0-703e-4729-a99e-b5c701eb7c00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:53.946312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1231/2462] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:38:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:53.956694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:53.958205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:54.017822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9b1a63-f8eb-44c3-8e49-0a615e25243d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:54.021538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06030f4f-3475-4176-80c0-44ad25b3b4db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:54.022191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06030f4f-3475-4176-80c0-44ad25b3b4db service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.022466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06030f4f-3475-4176-80c0-44ad25b3b4db service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.022739 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06030f4f-3475-4176-80c0-44ad25b3b4db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.032334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06030f4f-3475-4176-80c0-44ad25b3b4db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:54.033774 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06030f4f-3475-4176-80c0-44ad25b3b4db service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:54.046275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1232/2463] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:38:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:38:54.326973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91ea7e0-703e-4729-a99e-b5c701eb7c00 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:54.328501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d91ea7e0-703e-4729-a99e-b5c701eb7c00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:54.343823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91ea7e0-703e-4729-a99e-b5c701eb7c00 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:54.349524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:54.350723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.351359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.351631 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] PATH_INFO: `/v3/projects/4edd6fb784c14fd584ebe852a053526e` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.352992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4edd6fb784c14fd584ebe852a053526e)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:54.354846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:54.406283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4edd6fb784c14fd584ebe852a053526e', 'request_id': 'req-b08c1909-ab5e-49ea-b230-1add7b151d19'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:54.412177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4edd6fb784c14fd584ebe852a053526e', 'request_id': 'req-b08c1909-ab5e-49ea-b230-1add7b151d19'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:54.421334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4edd6fb784c14fd584ebe852a053526e', 'request_id': 'req-b08c1909-ab5e-49ea-b230-1add7b151d19'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:54.433261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] The token cache is being invalidate because project 4edd6fb784c14fd584ebe852a053526e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:38:54.433432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:54.434132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b08c1909-ab5e-49ea-b230-1add7b151d19 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:38:54.435641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1232/2464] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:38:54 2022] DELETE /identity/v3/projects/4edd6fb784c14fd584ebe852a053526e => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:38:54.441665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8086b-ecd9-4efb-8e5d-b204277eff0e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.442145 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8086b-ecd9-4efb-8e5d-b204277eff0e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.442483 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8086b-ecd9-4efb-8e5d-b204277eff0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.472517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-03c8086b-ecd9-4efb-8e5d-b204277eff0e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:54.473719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-03c8086b-ecd9-4efb-8e5d-b204277eff0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:54.478305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:54.539392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1233/2465] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:38:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:54.547444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ededefa-fe59-4ea4-b6fa-fdf017ea4915 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.547727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ededefa-fe59-4ea4-b6fa-fdf017ea4915 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.548001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ededefa-fe59-4ea4-b6fa-fdf017ea4915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.568121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3ededefa-fe59-4ea4-b6fa-fdf017ea4915 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:54.569324 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3ededefa-fe59-4ea4-b6fa-fdf017ea4915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:54.578767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1233/2466] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:38:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:54.583248 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c8086b-ecd9-4efb-8e5d-b204277eff0e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:54.593873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c8086b-ecd9-4efb-8e5d-b204277eff0e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:54.599261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19faf407-fde5-457d-a4b5-80bf6d548c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:54.599853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19faf407-fde5-457d-a4b5-80bf6d548c12 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.600121 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19faf407-fde5-457d-a4b5-80bf6d548c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.600383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19faf407-fde5-457d-a4b5-80bf6d548c12 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.601013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19faf407-fde5-457d-a4b5-80bf6d548c12 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:38:54.601308 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19faf407-fde5-457d-a4b5-80bf6d548c12 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:54.602905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19faf407-fde5-457d-a4b5-80bf6d548c12 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:54.615215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1234/2467] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:38:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:38:54.619701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ededefa-fe59-4ea4-b6fa-fdf017ea4915 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:54.629132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ededefa-fe59-4ea4-b6fa-fdf017ea4915 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:54.635256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7231c96-21f2-4c88-ab49-e14af4223e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:54.635861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7231c96-21f2-4c88-ab49-e14af4223e83 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.636135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7231c96-21f2-4c88-ab49-e14af4223e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.636396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7231c96-21f2-4c88-ab49-e14af4223e83 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.637033 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7231c96-21f2-4c88-ab49-e14af4223e83 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:54.639043 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7231c96-21f2-4c88-ab49-e14af4223e83 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:54.671362 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1234/2468] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:38:54 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:38:54.675962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19faf407-fde5-457d-a4b5-80bf6d548c12 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:54.686125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19faf407-fde5-457d-a4b5-80bf6d548c12 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:54.751308 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f7488e-3179-44d5-ae5e-b9c6c2fad14d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:54.752056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f7488e-3179-44d5-ae5e-b9c6c2fad14d admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.752298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f7488e-3179-44d5-ae5e-b9c6c2fad14d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.752596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f7488e-3179-44d5-ae5e-b9c6c2fad14d admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.753228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f7488e-3179-44d5-ae5e-b9c6c2fad14d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:54.755194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f7488e-3179-44d5-ae5e-b9c6c2fad14d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:54.778891 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1235/2469] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:38:54 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:38:54.783102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7231c96-21f2-4c88-ab49-e14af4223e83 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:54.793988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7231c96-21f2-4c88-ab49-e14af4223e83 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:54.799227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0398fe9-4066-405e-a662-ef5339db505f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:54.799853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0398fe9-4066-405e-a662-ef5339db505f admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.800124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0398fe9-4066-405e-a662-ef5339db505f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.800394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0398fe9-4066-405e-a662-ef5339db505f admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.801083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0398fe9-4066-405e-a662-ef5339db505f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:54.802748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0398fe9-4066-405e-a662-ef5339db505f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:54.813047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1235/2470] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:38:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:38:54.817195 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f7488e-3179-44d5-ae5e-b9c6c2fad14d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:54.828305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f7488e-3179-44d5-ae5e-b9c6c2fad14d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:54.834026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351ebf9a-df60-439f-93fd-74b7221ebe01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:54.834686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351ebf9a-df60-439f-93fd-74b7221ebe01 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.834963 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351ebf9a-df60-439f-93fd-74b7221ebe01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.835231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351ebf9a-df60-439f-93fd-74b7221ebe01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.836543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351ebf9a-df60-439f-93fd-74b7221ebe01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:54.838163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351ebf9a-df60-439f-93fd-74b7221ebe01 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:54.849658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1236/2471] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:38:54 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:38:54.854040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0398fe9-4066-405e-a662-ef5339db505f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:54.864425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0398fe9-4066-405e-a662-ef5339db505f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:54.869143 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d144ff8f-3bb7-49af-9d5d-7431cb141c93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:54.869761 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d144ff8f-3bb7-49af-9d5d-7431cb141c93 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.870073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d144ff8f-3bb7-49af-9d5d-7431cb141c93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.870376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d144ff8f-3bb7-49af-9d5d-7431cb141c93 admin admin] PATH_INFO: `/v3/projects/7eb316ace93e48c68ade74e9c5fccbf8/users/cb98650b97664e0cbac0f574163b8c40/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.884792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d144ff8f-3bb7-49af-9d5d-7431cb141c93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7eb316ace93e48c68ade74e9c5fccbf8, user_id=cb98650b97664e0cbac0f574163b8c40, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:54.887523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d144ff8f-3bb7-49af-9d5d-7431cb141c93 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:54.902462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1236/2472] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:38:54 2022] PUT /identity/v3/projects/7eb316ace93e48c68ade74e9c5fccbf8/users/cb98650b97664e0cbac0f574163b8c40/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:38:54.907799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0e2c10-8f80-4814-8b76-f7fc08ea7d7e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:54.908217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0e2c10-8f80-4814-8b76-f7fc08ea7d7e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:54.908493 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0e2c10-8f80-4814-8b76-f7fc08ea7d7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:54.928590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-cf0e2c10-8f80-4814-8b76-f7fc08ea7d7e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:54.929809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-cf0e2c10-8f80-4814-8b76-f7fc08ea7d7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:54.991677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1237/2473] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:38:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:55.001827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d144ff8f-3bb7-49af-9d5d-7431cb141c93 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:55.003654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d144ff8f-3bb7-49af-9d5d-7431cb141c93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:55.014469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:55.066026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d144ff8f-3bb7-49af-9d5d-7431cb141c93 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:55.070662 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c92cc4-8560-4bc8-92fc-b66af1670180 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:55.071302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c92cc4-8560-4bc8-92fc-b66af1670180 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:55.071584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c92cc4-8560-4bc8-92fc-b66af1670180 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:55.071860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c92cc4-8560-4bc8-92fc-b66af1670180 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:55.082777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c92cc4-8560-4bc8-92fc-b66af1670180 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:55.084230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c92cc4-8560-4bc8-92fc-b66af1670180 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:55.094129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1237/2474] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:38:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:38:59.307484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75105521-5ced-4c28-a3da-a4d7c2de8d1b None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:59.308001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75105521-5ced-4c28-a3da-a4d7c2de8d1b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:59.308001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75105521-5ced-4c28-a3da-a4d7c2de8d1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:59.336316 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-75105521-5ced-4c28-a3da-a4d7c2de8d1b None None] MFA Rules not processed for user `cb98650b97664e0cbac0f574163b8c40`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:38:59.337521 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-75105521-5ced-4c28-a3da-a4d7c2de8d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:38:59.341960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:38:59.406676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1238/2475] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:38:59 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:38:59.411539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c92cc4-8560-4bc8-92fc-b66af1670180 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:59.421241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c92cc4-8560-4bc8-92fc-b66af1670180 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:59.426628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d71e251-152c-4b29-ba4c-d03b434b5c75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:59.427243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d71e251-152c-4b29-ba4c-d03b434b5c75 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:59.427519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d71e251-152c-4b29-ba4c-d03b434b5c75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:59.427788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d71e251-152c-4b29-ba4c-d03b434b5c75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:59.428432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d71e251-152c-4b29-ba4c-d03b434b5c75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:59.430393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d71e251-152c-4b29-ba4c-d03b434b5c75 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:59.464451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1238/2476] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:38:59 2022] POST /identity/v3/projects => generated 371 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:38:59.468905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75105521-5ced-4c28-a3da-a4d7c2de8d1b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:59.481010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75105521-5ced-4c28-a3da-a4d7c2de8d1b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:59.560843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d77170-cf97-46cd-a582-591095664fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:59.561474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d77170-cf97-46cd-a582-591095664fe0 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:59.561750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d77170-cf97-46cd-a582-591095664fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:59.562015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d77170-cf97-46cd-a582-591095664fe0 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:59.562682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d77170-cf97-46cd-a582-591095664fe0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:59.564789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d77170-cf97-46cd-a582-591095664fe0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:59.592012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1239/2477] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:38:59 2022] POST /identity/v3/users => generated 292 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:38:59.593579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d71e251-152c-4b29-ba4c-d03b434b5c75 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:59.605347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d71e251-152c-4b29-ba4c-d03b434b5c75 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:59.609834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fab743-9e25-412c-86a9-d6a02aa8a88a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:59.610637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fab743-9e25-412c-86a9-d6a02aa8a88a admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:59.610949 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fab743-9e25-412c-86a9-d6a02aa8a88a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:59.611229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fab743-9e25-412c-86a9-d6a02aa8a88a admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:59.611858 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fab743-9e25-412c-86a9-d6a02aa8a88a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:59.613471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fab743-9e25-412c-86a9-d6a02aa8a88a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:59.627617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1239/2478] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:38:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:38:59.634423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d77170-cf97-46cd-a582-591095664fe0 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:59.644992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d77170-cf97-46cd-a582-591095664fe0 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:59.650315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6d7301-4f4c-4535-b1d1-3589f7701f6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:59.651541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6d7301-4f4c-4535-b1d1-3589f7701f6a admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:59.651541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6d7301-4f4c-4535-b1d1-3589f7701f6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:59.651790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6d7301-4f4c-4535-b1d1-3589f7701f6a admin admin] PATH_INFO: `/v3/projects/c4845b9e9baf48ffb49701d088fb2400/users/49d56d51ce70408ab780ef7c6faba482/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:59.668502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6d7301-4f4c-4535-b1d1-3589f7701f6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4845b9e9baf48ffb49701d088fb2400, user_id=49d56d51ce70408ab780ef7c6faba482, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:59.672668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6d7301-4f4c-4535-b1d1-3589f7701f6a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:59.688895 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1240/2479] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:38:59 2022] PUT /identity/v3/projects/c4845b9e9baf48ffb49701d088fb2400/users/49d56d51ce70408ab780ef7c6faba482/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:38:59.693061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fab743-9e25-412c-86a9-d6a02aa8a88a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:59.704653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fab743-9e25-412c-86a9-d6a02aa8a88a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:59.783081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7d128e-8935-47ad-8a6f-2becafe21bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:59.783698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7d128e-8935-47ad-8a6f-2becafe21bdb admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:59.784118 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7d128e-8935-47ad-8a6f-2becafe21bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:59.784394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7d128e-8935-47ad-8a6f-2becafe21bdb admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:59.785562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7d128e-8935-47ad-8a6f-2becafe21bdb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:59.787588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7d128e-8935-47ad-8a6f-2becafe21bdb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:59.800480 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1240/2480] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:38:59 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:38:59.804881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6d7301-4f4c-4535-b1d1-3589f7701f6a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:38:59.817021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6d7301-4f4c-4535-b1d1-3589f7701f6a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:38:59.829227 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952e7f90-6b39-4e02-8f09-97c2270beac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:38:59.829957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952e7f90-6b39-4e02-8f09-97c2270beac3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:38:59.830281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952e7f90-6b39-4e02-8f09-97c2270beac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:38:59.830571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952e7f90-6b39-4e02-8f09-97c2270beac3 admin admin] PATH_INFO: `/v3/projects/c4845b9e9baf48ffb49701d088fb2400/users/49d56d51ce70408ab780ef7c6faba482/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:38:59.832564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952e7f90-6b39-4e02-8f09-97c2270beac3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4845b9e9baf48ffb49701d088fb2400, user_id=49d56d51ce70408ab780ef7c6faba482, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:38:59.835872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952e7f90-6b39-4e02-8f09-97c2270beac3 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:38:59.853094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1241/2481] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:38:59 2022] PUT /identity/v3/projects/c4845b9e9baf48ffb49701d088fb2400/users/49d56d51ce70408ab780ef7c6faba482/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:05.446167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1181cea5-f56a-449a-beb3-2eb993253592 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:05.446784 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1181cea5-f56a-449a-beb3-2eb993253592 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:05.446963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1181cea5-f56a-449a-beb3-2eb993253592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:05.486510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-1181cea5-f56a-449a-beb3-2eb993253592 None None] MFA Rules not processed for user `49d56d51ce70408ab780ef7c6faba482`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:05.487772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1181cea5-f56a-449a-beb3-2eb993253592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:05.660909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:05.674249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1241/2482] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:05 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:05.724063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952e7f90-6b39-4e02-8f09-97c2270beac3 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:05.728277 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-952e7f90-6b39-4e02-8f09-97c2270beac3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:05.856484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:05.892836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952e7f90-6b39-4e02-8f09-97c2270beac3 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:05.898830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290338b6-93cd-445d-9f47-d751b1c736b3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:05.899681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290338b6-93cd-445d-9f47-d751b1c736b3 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:05.899681 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290338b6-93cd-445d-9f47-d751b1c736b3 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:05.899968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290338b6-93cd-445d-9f47-d751b1c736b3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:05.981449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290338b6-93cd-445d-9f47-d751b1c736b3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:05.983360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290338b6-93cd-445d-9f47-d751b1c736b3 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:06.000819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1242/2483] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:05 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:39:07.649221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea11194-8c9b-4cd7-80a5-6165f2a10f01 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:07.649834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea11194-8c9b-4cd7-80a5-6165f2a10f01 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:07.649834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea11194-8c9b-4cd7-80a5-6165f2a10f01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:07.671707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-eea11194-8c9b-4cd7-80a5-6165f2a10f01 None None] MFA Rules not processed for user `cb98650b97664e0cbac0f574163b8c40`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:07.673452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-eea11194-8c9b-4cd7-80a5-6165f2a10f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:07.685545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1242/2484] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:07 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:15.973667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290338b6-93cd-445d-9f47-d751b1c736b3 service nova] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:16.032511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290338b6-93cd-445d-9f47-d751b1c736b3 service nova] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:16.036790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4340a5a8-0668-4e0d-9348-cbca8c485399 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:16.037394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4340a5a8-0668-4e0d-9348-cbca8c485399 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:16.037663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4340a5a8-0668-4e0d-9348-cbca8c485399 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:16.037929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4340a5a8-0668-4e0d-9348-cbca8c485399 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:16.040132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4340a5a8-0668-4e0d-9348-cbca8c485399 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:16.110954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4340a5a8-0668-4e0d-9348-cbca8c485399 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:16.112385 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4340a5a8-0668-4e0d-9348-cbca8c485399 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:16.124826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1243/2485] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:39:17.667084 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:19.458383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea11194-8c9b-4cd7-80a5-6165f2a10f01 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:19.460645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-eea11194-8c9b-4cd7-80a5-6165f2a10f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:19.475434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea11194-8c9b-4cd7-80a5-6165f2a10f01 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:19.480924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:19.481598 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:19.481879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:19.482151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] PATH_INFO: `/v3/users/c02aec8271af4bf7aea728ff92fa555b` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:19.483444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c02aec8271af4bf7aea728ff92fa555b)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:19.485395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:19.542457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c02aec8271af4bf7aea728ff92fa555b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:19.543319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c02aec8271af4bf7aea728ff92fa555b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:19.551051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c02aec8271af4bf7aea728ff92fa555b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:19.551740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c02aec8271af4bf7aea728ff92fa555b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:19.560492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-6f428134-574e-4b68-8143-2e2d84bd3fa6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c02aec8271af4bf7aea728ff92fa555b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:19.579603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1243/2486] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:39:19 2022] DELETE /identity/v3/users/c02aec8271af4bf7aea728ff92fa555b => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:19.760848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4340a5a8-0668-4e0d-9348-cbca8c485399 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:19.772799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4340a5a8-0668-4e0d-9348-cbca8c485399 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:19.817214 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:19.833003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:19.833556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:19.833832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:19.834126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] PATH_INFO: `/v3/projects/597b9e023e324874ac5f24ee571d1f66` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:19.835263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] RBAC: Authorizing `identity:delete_project(project_id=597b9e023e324874ac5f24ee571d1f66)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:19.837206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:19.893830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '597b9e023e324874ac5f24ee571d1f66', 'request_id': 'req-13f49d00-54a2-48a0-aaca-c00e5147ae05'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:19.901013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '597b9e023e324874ac5f24ee571d1f66', 'request_id': 'req-13f49d00-54a2-48a0-aaca-c00e5147ae05'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:19.911776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '597b9e023e324874ac5f24ee571d1f66', 'request_id': 'req-13f49d00-54a2-48a0-aaca-c00e5147ae05'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:19.923363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] The token cache is being invalidate because project 597b9e023e324874ac5f24ee571d1f66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:39:19.923646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:19.924288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-13f49d00-54a2-48a0-aaca-c00e5147ae05 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:19.925865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1244/2487] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:39:19 2022] DELETE /identity/v3/projects/597b9e023e324874ac5f24ee571d1f66 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:19.931124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0acd8c-6111-4525-8696-d96a7d555d38 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:19.931407 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0acd8c-6111-4525-8696-d96a7d555d38 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:19.931672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0acd8c-6111-4525-8696-d96a7d555d38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:19.952204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7b0acd8c-6111-4525-8696-d96a7d555d38 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:19.953390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7b0acd8c-6111-4525-8696-d96a7d555d38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:20.025937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1244/2488] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:20.035017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e933ac-f667-4a4b-9925-577fefdba4d3 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.035671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e933ac-f667-4a4b-9925-577fefdba4d3 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.035671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e933ac-f667-4a4b-9925-577fefdba4d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.056685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d8e933ac-f667-4a4b-9925-577fefdba4d3 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:20.057904 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d8e933ac-f667-4a4b-9925-577fefdba4d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:20.068231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1245/2489] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:20.072770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0acd8c-6111-4525-8696-d96a7d555d38 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.083116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0acd8c-6111-4525-8696-d96a7d555d38 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.087687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0761c5a-840c-487e-bbf3-e16969f3dbae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.088296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0761c5a-840c-487e-bbf3-e16969f3dbae admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.088566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0761c5a-840c-487e-bbf3-e16969f3dbae admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.088837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0761c5a-840c-487e-bbf3-e16969f3dbae admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.089471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0761c5a-840c-487e-bbf3-e16969f3dbae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:39:20.089769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0761c5a-840c-487e-bbf3-e16969f3dbae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.091405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0761c5a-840c-487e-bbf3-e16969f3dbae admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.103435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1245/2490] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:39:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:39:20.107851 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e933ac-f667-4a4b-9925-577fefdba4d3 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.117589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e933ac-f667-4a4b-9925-577fefdba4d3 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.123065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2651b5ce-7307-4293-8446-5bdec6ef9ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.123688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2651b5ce-7307-4293-8446-5bdec6ef9ad4 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.123957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2651b5ce-7307-4293-8446-5bdec6ef9ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.124219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2651b5ce-7307-4293-8446-5bdec6ef9ad4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.124846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2651b5ce-7307-4293-8446-5bdec6ef9ad4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.126762 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2651b5ce-7307-4293-8446-5bdec6ef9ad4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.159085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1246/2491] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:39:20 2022] POST /identity/v3/projects => generated 355 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:39:20.165990 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0761c5a-840c-487e-bbf3-e16969f3dbae admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.179819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0761c5a-840c-487e-bbf3-e16969f3dbae admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.239107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a019dd-2443-45b3-8be4-65d542152a78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.239709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a019dd-2443-45b3-8be4-65d542152a78 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.239987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a019dd-2443-45b3-8be4-65d542152a78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.240257 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a019dd-2443-45b3-8be4-65d542152a78 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.240867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a019dd-2443-45b3-8be4-65d542152a78 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.242848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a019dd-2443-45b3-8be4-65d542152a78 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.250979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:20.269085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1246/2492] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:39:20 2022] POST /identity/v3/users => generated 278 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:39:20.273555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2651b5ce-7307-4293-8446-5bdec6ef9ad4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.283866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2651b5ce-7307-4293-8446-5bdec6ef9ad4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.289180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914fbcf3-b8f0-4a05-b10b-28f6f6087d60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.289815 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914fbcf3-b8f0-4a05-b10b-28f6f6087d60 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.290091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914fbcf3-b8f0-4a05-b10b-28f6f6087d60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.290418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914fbcf3-b8f0-4a05-b10b-28f6f6087d60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.291044 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914fbcf3-b8f0-4a05-b10b-28f6f6087d60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.292647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914fbcf3-b8f0-4a05-b10b-28f6f6087d60 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.302816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1247/2493] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:20.306847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a019dd-2443-45b3-8be4-65d542152a78 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.316918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a019dd-2443-45b3-8be4-65d542152a78 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.321653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3567a0-6115-4be3-bd2e-990130c73188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.322285 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3567a0-6115-4be3-bd2e-990130c73188 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.322568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3567a0-6115-4be3-bd2e-990130c73188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.322838 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3567a0-6115-4be3-bd2e-990130c73188 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.323475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3567a0-6115-4be3-bd2e-990130c73188 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.325412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3567a0-6115-4be3-bd2e-990130c73188 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.336123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1247/2494] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:20.340077 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914fbcf3-b8f0-4a05-b10b-28f6f6087d60 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.350479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914fbcf3-b8f0-4a05-b10b-28f6f6087d60 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.355412 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2bd12f-4e2a-4d07-8ea2-9c1a678f7efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.356063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2bd12f-4e2a-4d07-8ea2-9c1a678f7efb admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.356341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2bd12f-4e2a-4d07-8ea2-9c1a678f7efb admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.356634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2bd12f-4e2a-4d07-8ea2-9c1a678f7efb admin admin] PATH_INFO: `/v3/projects/48a39b674d59433b8767d81db002ad8d/users/df5325a918484940a685612f41bcee36/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.372209 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2bd12f-4e2a-4d07-8ea2-9c1a678f7efb admin admin] RBAC: Authorizing `identity:create_grant(project_id=48a39b674d59433b8767d81db002ad8d, user_id=df5325a918484940a685612f41bcee36, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.374888 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2bd12f-4e2a-4d07-8ea2-9c1a678f7efb admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.389632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1248/2495] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:39:20 2022] PUT /identity/v3/projects/48a39b674d59433b8767d81db002ad8d/users/df5325a918484940a685612f41bcee36/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:20.401431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f488db-a6a2-4021-bffd-572c99fcae5b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.401716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f488db-a6a2-4021-bffd-572c99fcae5b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.401990 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f488db-a6a2-4021-bffd-572c99fcae5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.430811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-46f488db-a6a2-4021-bffd-572c99fcae5b None None] MFA Rules not processed for user `df5325a918484940a685612f41bcee36`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:20.432032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-46f488db-a6a2-4021-bffd-572c99fcae5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:20.501887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1248/2496] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:20 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:20.507012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2bd12f-4e2a-4d07-8ea2-9c1a678f7efb admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.575072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2bd12f-4e2a-4d07-8ea2-9c1a678f7efb admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.580547 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444510de-ca1b-45f0-9a6f-318d69253eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.581116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444510de-ca1b-45f0-9a6f-318d69253eb5 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.581409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444510de-ca1b-45f0-9a6f-318d69253eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.581667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444510de-ca1b-45f0-9a6f-318d69253eb5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.582557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444510de-ca1b-45f0-9a6f-318d69253eb5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.584465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444510de-ca1b-45f0-9a6f-318d69253eb5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.599692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:20.615902 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1249/2497] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:39:20 2022] POST /identity/v3/projects => generated 355 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:39:20.620316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f488db-a6a2-4021-bffd-572c99fcae5b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.631528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f488db-a6a2-4021-bffd-572c99fcae5b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.698594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c8f69e-617d-48a6-a62c-32100f90a765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.699559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c8f69e-617d-48a6-a62c-32100f90a765 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.699835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c8f69e-617d-48a6-a62c-32100f90a765 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.700105 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c8f69e-617d-48a6-a62c-32100f90a765 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.700739 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c8f69e-617d-48a6-a62c-32100f90a765 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.702724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c8f69e-617d-48a6-a62c-32100f90a765 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.726971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1249/2498] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:39:20 2022] POST /identity/v3/users => generated 284 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:39:20.731167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444510de-ca1b-45f0-9a6f-318d69253eb5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.741621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444510de-ca1b-45f0-9a6f-318d69253eb5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.747015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cda733a-740e-44ea-8795-62614dbeccb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.747623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cda733a-740e-44ea-8795-62614dbeccb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.747899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cda733a-740e-44ea-8795-62614dbeccb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.748164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cda733a-740e-44ea-8795-62614dbeccb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.748802 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cda733a-740e-44ea-8795-62614dbeccb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.750441 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cda733a-740e-44ea-8795-62614dbeccb5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.761920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1250/2499] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:20.766333 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c8f69e-617d-48a6-a62c-32100f90a765 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.777520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c8f69e-617d-48a6-a62c-32100f90a765 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.783353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d9845-aca0-4550-9104-12e4c3a1b683 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.783998 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d9845-aca0-4550-9104-12e4c3a1b683 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.784272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d9845-aca0-4550-9104-12e4c3a1b683 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.784535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d9845-aca0-4550-9104-12e4c3a1b683 admin admin] PATH_INFO: `/v3/projects/26aebc6cd87545efb0fde8df7afc343e/users/c4eb1fa8294f42668f9585ed7979042d/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.800748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206d9845-aca0-4550-9104-12e4c3a1b683 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26aebc6cd87545efb0fde8df7afc343e, user_id=c4eb1fa8294f42668f9585ed7979042d, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.803453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206d9845-aca0-4550-9104-12e4c3a1b683 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.819254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1250/2500] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:39:20 2022] PUT /identity/v3/projects/26aebc6cd87545efb0fde8df7afc343e/users/c4eb1fa8294f42668f9585ed7979042d/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:39:20.824242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cda733a-740e-44ea-8795-62614dbeccb5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.834672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cda733a-740e-44ea-8795-62614dbeccb5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.899196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1eee35f-4dee-4e60-a47c-b04bdae83916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.899791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eee35f-4dee-4e60-a47c-b04bdae83916 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.900063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eee35f-4dee-4e60-a47c-b04bdae83916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.900319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eee35f-4dee-4e60-a47c-b04bdae83916 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.900959 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1eee35f-4dee-4e60-a47c-b04bdae83916 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.902594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1eee35f-4dee-4e60-a47c-b04bdae83916 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.912838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1251/2501] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:20 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:20.917352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d9845-aca0-4550-9104-12e4c3a1b683 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:20.928643 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d9845-aca0-4550-9104-12e4c3a1b683 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:20.933646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7aa8b8-6650-46bc-b7d2-99e3f0d9b3a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:20.934345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7aa8b8-6650-46bc-b7d2-99e3f0d9b3a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.934617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7aa8b8-6650-46bc-b7d2-99e3f0d9b3a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.935003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7aa8b8-6650-46bc-b7d2-99e3f0d9b3a9 admin admin] PATH_INFO: `/v3/projects/26aebc6cd87545efb0fde8df7afc343e/users/c4eb1fa8294f42668f9585ed7979042d/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.936974 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7aa8b8-6650-46bc-b7d2-99e3f0d9b3a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26aebc6cd87545efb0fde8df7afc343e, user_id=c4eb1fa8294f42668f9585ed7979042d, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:20.939632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7aa8b8-6650-46bc-b7d2-99e3f0d9b3a9 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:20.950896 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1251/2502] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:39:20 2022] PUT /identity/v3/projects/26aebc6cd87545efb0fde8df7afc343e/users/c4eb1fa8294f42668f9585ed7979042d/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:20.961816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475614bd-ed3e-4f9a-8f0b-6d08f98f5f27 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:20.962137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475614bd-ed3e-4f9a-8f0b-6d08f98f5f27 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:20.962409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475614bd-ed3e-4f9a-8f0b-6d08f98f5f27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:20.990808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-475614bd-ed3e-4f9a-8f0b-6d08f98f5f27 None None] MFA Rules not processed for user `c4eb1fa8294f42668f9585ed7979042d`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:20.992014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-475614bd-ed3e-4f9a-8f0b-6d08f98f5f27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:21.064508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1252/2503] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:20 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:21.077545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7aa8b8-6650-46bc-b7d2-99e3f0d9b3a9 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:21.079691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7f7aa8b8-6650-46bc-b7d2-99e3f0d9b3a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:21.105602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:21.167097 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7aa8b8-6650-46bc-b7d2-99e3f0d9b3a9 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:21.171373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c281b4-e595-4d39-9709-e1489384e360 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:21.171956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c281b4-e595-4d39-9709-e1489384e360 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:21.172235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c281b4-e595-4d39-9709-e1489384e360 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:21.172510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c281b4-e595-4d39-9709-e1489384e360 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:21.183908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c281b4-e595-4d39-9709-e1489384e360 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:21.185691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c281b4-e595-4d39-9709-e1489384e360 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:21.196085 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1252/2504] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:21 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:39:21.279567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475614bd-ed3e-4f9a-8f0b-6d08f98f5f27 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:21.290587 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475614bd-ed3e-4f9a-8f0b-6d08f98f5f27 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:21.294637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821cb00b-3daa-4073-98d0-29d63b2a723d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:21.295243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821cb00b-3daa-4073-98d0-29d63b2a723d service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:21.295517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821cb00b-3daa-4073-98d0-29d63b2a723d service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:21.295803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821cb00b-3daa-4073-98d0-29d63b2a723d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:21.355381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821cb00b-3daa-4073-98d0-29d63b2a723d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:21.356826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821cb00b-3daa-4073-98d0-29d63b2a723d service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:21.366541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1253/2505] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:21 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:39:31.191170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=26 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:31.361606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=98 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:37.698507 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c281b4-e595-4d39-9709-e1489384e360 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:37.758815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c281b4-e595-4d39-9709-e1489384e360 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:37.763657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:37.764272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:37.764554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:37.764815 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] PATH_INFO: `/v3/users/df5325a918484940a685612f41bcee36` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:37.765934 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df5325a918484940a685612f41bcee36)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:37.767884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:37.819940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df5325a918484940a685612f41bcee36'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:37.820603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df5325a918484940a685612f41bcee36'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:37.828383 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df5325a918484940a685612f41bcee36'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:37.828991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df5325a918484940a685612f41bcee36'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:37.840071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df5325a918484940a685612f41bcee36'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:37.862124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1253/2506] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:39:37 2022] DELETE /identity/v3/users/df5325a918484940a685612f41bcee36 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:37.867114 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaea74d-b864-4991-84c1-22b480626175 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:37.867640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaea74d-b864-4991-84c1-22b480626175 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:37.867640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaea74d-b864-4991-84c1-22b480626175 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:37.887834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-2eaea74d-b864-4991-84c1-22b480626175 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:37.889005 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2eaea74d-b864-4991-84c1-22b480626175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:37.950797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1254/2507] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:37.960905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:37.962456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:38.023053 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d52934-2559-45b3-8090-2d4fd1cd9051 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:38.033603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bb5abc-2b41-47c3-985e-76ec5351323b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:38.034267 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bb5abc-2b41-47c3-985e-76ec5351323b service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:38.034561 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bb5abc-2b41-47c3-985e-76ec5351323b service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:38.034837 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bb5abc-2b41-47c3-985e-76ec5351323b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:38.046417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bb5abc-2b41-47c3-985e-76ec5351323b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:38.047874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bb5abc-2b41-47c3-985e-76ec5351323b service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:38.058602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1254/2508] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:39:38.361030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaea74d-b864-4991-84c1-22b480626175 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:38.362615 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2eaea74d-b864-4991-84c1-22b480626175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:38.375109 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaea74d-b864-4991-84c1-22b480626175 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:38.380389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:38.381435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:38.381818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:38.382163 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] PATH_INFO: `/v3/projects/48a39b674d59433b8767d81db002ad8d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:38.383315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=48a39b674d59433b8767d81db002ad8d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:38.385267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:38.434129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48a39b674d59433b8767d81db002ad8d', 'request_id': 'req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:38.439816 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48a39b674d59433b8767d81db002ad8d', 'request_id': 'req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:38.449349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48a39b674d59433b8767d81db002ad8d', 'request_id': 'req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:38.461392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] The token cache is being invalidate because project 48a39b674d59433b8767d81db002ad8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:39:38.461677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:38.462395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:38.463842 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1255/2509] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:39:38 2022] DELETE /identity/v3/projects/48a39b674d59433b8767d81db002ad8d => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:38.470356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bb5abc-2b41-47c3-985e-76ec5351323b service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:38.480951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bb5abc-2b41-47c3-985e-76ec5351323b service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:38.499680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:38.543022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:38.543666 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:38.543933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:38.544205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] PATH_INFO: `/v3/users/c4eb1fa8294f42668f9585ed7979042d` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:38.545399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4eb1fa8294f42668f9585ed7979042d)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:38.547393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:38.597660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4eb1fa8294f42668f9585ed7979042d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:38.598319 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4eb1fa8294f42668f9585ed7979042d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:38.605045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4eb1fa8294f42668f9585ed7979042d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:38.605655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4eb1fa8294f42668f9585ed7979042d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:38.617885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-88998e3a-be13-41bc-ae66-5cfd88051906 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4eb1fa8294f42668f9585ed7979042d'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:38.640844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1255/2510] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:39:38 2022] DELETE /identity/v3/users/c4eb1fa8294f42668f9585ed7979042d => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:38.912917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:38.914548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:38.927733 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3dc56a-3da5-4b53-bce7-1f04cbbf70d7 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:38.991266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:38.991810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:38.992086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:38.992353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] PATH_INFO: `/v3/projects/26aebc6cd87545efb0fde8df7afc343e` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:38.993491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=26aebc6cd87545efb0fde8df7afc343e)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:38.995425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.027524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:39.045366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26aebc6cd87545efb0fde8df7afc343e', 'request_id': 'req-fb687d76-6322-4a50-acae-aaebbace30cd'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:39.050897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26aebc6cd87545efb0fde8df7afc343e', 'request_id': 'req-fb687d76-6322-4a50-acae-aaebbace30cd'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:39.061128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26aebc6cd87545efb0fde8df7afc343e', 'request_id': 'req-fb687d76-6322-4a50-acae-aaebbace30cd'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:39.074340 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] The token cache is being invalidate because project 26aebc6cd87545efb0fde8df7afc343e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:39:39.074625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:39.075398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-fb687d76-6322-4a50-acae-aaebbace30cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:39.077070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1256/2511] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:39:38 2022] DELETE /identity/v3/projects/26aebc6cd87545efb0fde8df7afc343e => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:39.083237 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d144a2-73ce-4345-b2ea-1b86d999d067 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.083771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d144a2-73ce-4345-b2ea-1b86d999d067 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.083771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d144a2-73ce-4345-b2ea-1b86d999d067 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.110688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-f7d144a2-73ce-4345-b2ea-1b86d999d067 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:39.111861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f7d144a2-73ce-4345-b2ea-1b86d999d067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:39.172405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1256/2512] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:39.181937 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b8bb7-9fcc-4855-ac0c-1eb1034ae178 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.182254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b8bb7-9fcc-4855-ac0c-1eb1034ae178 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.182525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b8bb7-9fcc-4855-ac0c-1eb1034ae178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.203148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b47b8bb7-9fcc-4855-ac0c-1eb1034ae178 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:39.204332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b47b8bb7-9fcc-4855-ac0c-1eb1034ae178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:39.212999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1257/2513] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:39.217854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d144a2-73ce-4345-b2ea-1b86d999d067 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:39.231458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d144a2-73ce-4345-b2ea-1b86d999d067 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:39.237860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419058c3-e641-4561-bc23-28160dd78a33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:39.238499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419058c3-e641-4561-bc23-28160dd78a33 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.238779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419058c3-e641-4561-bc23-28160dd78a33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.239045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419058c3-e641-4561-bc23-28160dd78a33 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.239698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419058c3-e641-4561-bc23-28160dd78a33 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:39:39.240021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419058c3-e641-4561-bc23-28160dd78a33 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:39.241692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419058c3-e641-4561-bc23-28160dd78a33 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.254921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1257/2514] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:39:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:39:39.260860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47b8bb7-9fcc-4855-ac0c-1eb1034ae178 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:39.274432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47b8bb7-9fcc-4855-ac0c-1eb1034ae178 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:39.281349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32eeaf62-10aa-4f33-850c-a71726a30bb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:39.282225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eeaf62-10aa-4f33-850c-a71726a30bb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.282495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eeaf62-10aa-4f33-850c-a71726a30bb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.282760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32eeaf62-10aa-4f33-850c-a71726a30bb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.283395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32eeaf62-10aa-4f33-850c-a71726a30bb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:39.285903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32eeaf62-10aa-4f33-850c-a71726a30bb4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.320786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1258/2515] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:39:39 2022] POST /identity/v3/projects => generated 363 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:39:39.325531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419058c3-e641-4561-bc23-28160dd78a33 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:39.336220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419058c3-e641-4561-bc23-28160dd78a33 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:39.362394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:39.396280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4062724-a1b8-48c6-9456-402daf08995e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:39.396962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4062724-a1b8-48c6-9456-402daf08995e admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.397236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4062724-a1b8-48c6-9456-402daf08995e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.397493 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4062724-a1b8-48c6-9456-402daf08995e admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.398142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4062724-a1b8-48c6-9456-402daf08995e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:39.400089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4062724-a1b8-48c6-9456-402daf08995e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.422213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1258/2516] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:39:39 2022] POST /identity/v3/users => generated 282 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:39:39.426555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32eeaf62-10aa-4f33-850c-a71726a30bb4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:39.442966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32eeaf62-10aa-4f33-850c-a71726a30bb4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:39.452675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746426af-9699-4812-a3b6-c5d86dfa6be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:39.453558 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746426af-9699-4812-a3b6-c5d86dfa6be5 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.454001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746426af-9699-4812-a3b6-c5d86dfa6be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.454479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746426af-9699-4812-a3b6-c5d86dfa6be5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.455478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746426af-9699-4812-a3b6-c5d86dfa6be5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:39.458054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746426af-9699-4812-a3b6-c5d86dfa6be5 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.468996 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1259/2517] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:39 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:39.473248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4062724-a1b8-48c6-9456-402daf08995e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:39.484282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4062724-a1b8-48c6-9456-402daf08995e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:39.489409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c309aa-dcfe-4dae-a104-27e53eb2196d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:39.490003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c309aa-dcfe-4dae-a104-27e53eb2196d admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.490401 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c309aa-dcfe-4dae-a104-27e53eb2196d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.490675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c309aa-dcfe-4dae-a104-27e53eb2196d admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.491326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c309aa-dcfe-4dae-a104-27e53eb2196d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:39.493081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c309aa-dcfe-4dae-a104-27e53eb2196d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.503215 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1259/2518] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:39.507751 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746426af-9699-4812-a3b6-c5d86dfa6be5 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:39.517910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746426af-9699-4812-a3b6-c5d86dfa6be5 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:39.524054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e53df0-b75c-4a83-9b91-5b7a4aca03a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:39.524700 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e53df0-b75c-4a83-9b91-5b7a4aca03a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.524985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e53df0-b75c-4a83-9b91-5b7a4aca03a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.525231 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e53df0-b75c-4a83-9b91-5b7a4aca03a6 admin admin] PATH_INFO: `/v3/projects/3bc739c34f634c1fb4f8cbd2186dbaa7/users/21f5451bc4cf43c48539353199eb27a8/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.540468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e53df0-b75c-4a83-9b91-5b7a4aca03a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3bc739c34f634c1fb4f8cbd2186dbaa7, user_id=21f5451bc4cf43c48539353199eb27a8, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:39.543159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e53df0-b75c-4a83-9b91-5b7a4aca03a6 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.558987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1260/2519] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:39:39 2022] PUT /identity/v3/projects/3bc739c34f634c1fb4f8cbd2186dbaa7/users/21f5451bc4cf43c48539353199eb27a8/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:39.569491 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10933e28-65dd-40a4-a531-ea2f402d7015 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.569773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10933e28-65dd-40a4-a531-ea2f402d7015 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.570044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10933e28-65dd-40a4-a531-ea2f402d7015 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.600699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-10933e28-65dd-40a4-a531-ea2f402d7015 None None] MFA Rules not processed for user `21f5451bc4cf43c48539353199eb27a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:39.601931 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-10933e28-65dd-40a4-a531-ea2f402d7015 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:39.671757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1260/2520] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:39 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:39.676596 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e53df0-b75c-4a83-9b91-5b7a4aca03a6 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:39.752901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e53df0-b75c-4a83-9b91-5b7a4aca03a6 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:39.760304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774596bc-ba10-481b-a084-a4badcc2e85a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:39.760909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774596bc-ba10-481b-a084-a4badcc2e85a admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.761179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774596bc-ba10-481b-a084-a4badcc2e85a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.761442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774596bc-ba10-481b-a084-a4badcc2e85a admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.762051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774596bc-ba10-481b-a084-a4badcc2e85a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:39.763990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774596bc-ba10-481b-a084-a4badcc2e85a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.794470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1261/2521] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:39:39 2022] POST /identity/v3/projects => generated 363 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:39:39.799012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10933e28-65dd-40a4-a531-ea2f402d7015 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:39.810024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10933e28-65dd-40a4-a531-ea2f402d7015 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:39.869312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8951a6-b951-470a-a2ec-956f2cba92d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:39.869972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8951a6-b951-470a-a2ec-956f2cba92d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.870296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8951a6-b951-470a-a2ec-956f2cba92d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.870583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8951a6-b951-470a-a2ec-956f2cba92d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.871293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8951a6-b951-470a-a2ec-956f2cba92d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:39.873341 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8951a6-b951-470a-a2ec-956f2cba92d1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.895622 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1261/2522] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:39:39 2022] POST /identity/v3/users => generated 288 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:39:39.900015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774596bc-ba10-481b-a084-a4badcc2e85a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:39.909823 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774596bc-ba10-481b-a084-a4badcc2e85a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:39.914645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3e3830-f56d-4987-acb6-d6e7e8299f7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:39.915258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3e3830-f56d-4987-acb6-d6e7e8299f7c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.915512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3e3830-f56d-4987-acb6-d6e7e8299f7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.915779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3e3830-f56d-4987-acb6-d6e7e8299f7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.916424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3e3830-f56d-4987-acb6-d6e7e8299f7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:39.918024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3e3830-f56d-4987-acb6-d6e7e8299f7c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.932382 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1262/2523] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:39 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:39.933722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8951a6-b951-470a-a2ec-956f2cba92d1 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:39.944597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8951a6-b951-470a-a2ec-956f2cba92d1 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:39.950338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123eed75-35fb-485e-bbb5-de3c01d5e3d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:39.951005 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123eed75-35fb-485e-bbb5-de3c01d5e3d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:39.951282 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123eed75-35fb-485e-bbb5-de3c01d5e3d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:39.951556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123eed75-35fb-485e-bbb5-de3c01d5e3d3 admin admin] PATH_INFO: `/v3/projects/02c11d71b8d540bb87a8261906076782/users/1ace729a2c874bb2a18d1673163edf9d/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:39.967018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123eed75-35fb-485e-bbb5-de3c01d5e3d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02c11d71b8d540bb87a8261906076782, user_id=1ace729a2c874bb2a18d1673163edf9d, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:39.969687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123eed75-35fb-485e-bbb5-de3c01d5e3d3 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:39.988819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1262/2524] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:39:39 2022] PUT /identity/v3/projects/02c11d71b8d540bb87a8261906076782/users/1ace729a2c874bb2a18d1673163edf9d/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:39.995045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3e3830-f56d-4987-acb6-d6e7e8299f7c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:40.009094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3e3830-f56d-4987-acb6-d6e7e8299f7c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:40.030001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:40.072288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a1a1fb-ff19-47a8-bfcf-eb7c332e270f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:40.073272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a1a1fb-ff19-47a8-bfcf-eb7c332e270f admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:40.073541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a1a1fb-ff19-47a8-bfcf-eb7c332e270f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:40.073808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a1a1fb-ff19-47a8-bfcf-eb7c332e270f admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:40.074501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a1a1fb-ff19-47a8-bfcf-eb7c332e270f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:40.076103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a1a1fb-ff19-47a8-bfcf-eb7c332e270f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:40.086814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1263/2525] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:39 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:40.091396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123eed75-35fb-485e-bbb5-de3c01d5e3d3 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:40.097049 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:40.102874 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123eed75-35fb-485e-bbb5-de3c01d5e3d3 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:40.108725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d543050b-d349-4a84-8e67-f0814d5d1aeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:40.109404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d543050b-d349-4a84-8e67-f0814d5d1aeb admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:40.109704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d543050b-d349-4a84-8e67-f0814d5d1aeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:40.109976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d543050b-d349-4a84-8e67-f0814d5d1aeb admin admin] PATH_INFO: `/v3/projects/02c11d71b8d540bb87a8261906076782/users/1ace729a2c874bb2a18d1673163edf9d/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:40.111960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d543050b-d349-4a84-8e67-f0814d5d1aeb admin admin] RBAC: Authorizing `identity:create_grant(project_id=02c11d71b8d540bb87a8261906076782, user_id=1ace729a2c874bb2a18d1673163edf9d, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:40.114665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d543050b-d349-4a84-8e67-f0814d5d1aeb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:40.125165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1263/2526] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:39:40 2022] PUT /identity/v3/projects/02c11d71b8d540bb87a8261906076782/users/1ace729a2c874bb2a18d1673163edf9d/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:40.135716 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc4105a-c51b-4e6b-a2ef-19cb226e62ea None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:40.135997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc4105a-c51b-4e6b-a2ef-19cb226e62ea None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:40.136262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc4105a-c51b-4e6b-a2ef-19cb226e62ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:40.165171 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8cc4105a-c51b-4e6b-a2ef-19cb226e62ea None None] MFA Rules not processed for user `1ace729a2c874bb2a18d1673163edf9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:40.166470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8cc4105a-c51b-4e6b-a2ef-19cb226e62ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:40.235695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1264/2527] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:40 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:40.247701 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d543050b-d349-4a84-8e67-f0814d5d1aeb admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:40.249740 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d543050b-d349-4a84-8e67-f0814d5d1aeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:40.313135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d543050b-d349-4a84-8e67-f0814d5d1aeb admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:40.316909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1c28c0-4a31-43f3-9cd5-d8c4ded41cb5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:40.317526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c28c0-4a31-43f3-9cd5-d8c4ded41cb5 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:40.317800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c28c0-4a31-43f3-9cd5-d8c4ded41cb5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:40.318072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c28c0-4a31-43f3-9cd5-d8c4ded41cb5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:40.327851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1c28c0-4a31-43f3-9cd5-d8c4ded41cb5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:40.329314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1c28c0-4a31-43f3-9cd5-d8c4ded41cb5 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:40.338827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1264/2528] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:40 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:39:40.421546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc4105a-c51b-4e6b-a2ef-19cb226e62ea None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:40.431830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc4105a-c51b-4e6b-a2ef-19cb226e62ea None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:40.435612 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a39cab9-ba51-46b9-a992-585194b4cce9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:40.436219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a39cab9-ba51-46b9-a992-585194b4cce9 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:40.436496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a39cab9-ba51-46b9-a992-585194b4cce9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:40.436768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a39cab9-ba51-46b9-a992-585194b4cce9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:40.499243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a39cab9-ba51-46b9-a992-585194b4cce9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:40.501275 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a39cab9-ba51-46b9-a992-585194b4cce9 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:40.512098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1265/2529] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:40 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:39:50.334821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:50.508004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=80 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:50.919155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1c28c0-4a31-43f3-9cd5-d8c4ded41cb5 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:50.920621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6e1c28c0-4a31-43f3-9cd5-d8c4ded41cb5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:50.977431 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1c28c0-4a31-43f3-9cd5-d8c4ded41cb5 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:50.981633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bc997f-b67a-424f-9dfb-48a3e19c53cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:50.982328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bc997f-b67a-424f-9dfb-48a3e19c53cb service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:50.982566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bc997f-b67a-424f-9dfb-48a3e19c53cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:50.982834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bc997f-b67a-424f-9dfb-48a3e19c53cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:50.984635 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a7bc997f-b67a-424f-9dfb-48a3e19c53cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:51.057335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bc997f-b67a-424f-9dfb-48a3e19c53cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:51.059168 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bc997f-b67a-424f-9dfb-48a3e19c53cb service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:51.069553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1265/2530] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:50 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:39:52.089683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a39cab9-ba51-46b9-a992-585194b4cce9 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:52.152969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a39cab9-ba51-46b9-a992-585194b4cce9 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:52.157841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:52.158525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:52.158817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:52.159204 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] PATH_INFO: `/v3/users/21f5451bc4cf43c48539353199eb27a8` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:52.160358 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=21f5451bc4cf43c48539353199eb27a8)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:52.162378 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:52.216595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21f5451bc4cf43c48539353199eb27a8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:52.228568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21f5451bc4cf43c48539353199eb27a8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:52.229496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21f5451bc4cf43c48539353199eb27a8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:52.230279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21f5451bc4cf43c48539353199eb27a8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:52.240900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21f5451bc4cf43c48539353199eb27a8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:52.263331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1266/2531] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:39:52 2022] DELETE /identity/v3/users/21f5451bc4cf43c48539353199eb27a8 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:52.267773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27755f94-c78d-44ee-ab8e-965d06ccd255 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:52.268274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27755f94-c78d-44ee-ab8e-965d06ccd255 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:52.268382 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27755f94-c78d-44ee-ab8e-965d06ccd255 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:52.289699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-27755f94-c78d-44ee-ab8e-965d06ccd255 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:52.291064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-27755f94-c78d-44ee-ab8e-965d06ccd255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:52.360328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1266/2532] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:52.374609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:52.376042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:52.451400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe0fbda-d28b-4f10-a55c-d29a415089cd admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:52.455127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9177ec85-4c16-42d7-a9d0-e5202b884877 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:52.455745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9177ec85-4c16-42d7-a9d0-e5202b884877 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:52.456017 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9177ec85-4c16-42d7-a9d0-e5202b884877 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:52.456287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9177ec85-4c16-42d7-a9d0-e5202b884877 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:52.468217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9177ec85-4c16-42d7-a9d0-e5202b884877 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:52.469667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9177ec85-4c16-42d7-a9d0-e5202b884877 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:52.479951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1267/2533] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:39:52.762535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27755f94-c78d-44ee-ab8e-965d06ccd255 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:52.764274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-27755f94-c78d-44ee-ab8e-965d06ccd255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:52.783723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27755f94-c78d-44ee-ab8e-965d06ccd255 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:52.796500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:52.797170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:52.797448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:52.797717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] PATH_INFO: `/v3/projects/3bc739c34f634c1fb4f8cbd2186dbaa7` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:52.798901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3bc739c34f634c1fb4f8cbd2186dbaa7)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:52.800835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:52.881250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:52.884788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3bc739c34f634c1fb4f8cbd2186dbaa7', 'request_id': 'req-3919a0ae-25e7-45cf-b89f-535790195679'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:52.893738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3bc739c34f634c1fb4f8cbd2186dbaa7', 'request_id': 'req-3919a0ae-25e7-45cf-b89f-535790195679'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:52.906384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3bc739c34f634c1fb4f8cbd2186dbaa7', 'request_id': 'req-3919a0ae-25e7-45cf-b89f-535790195679'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:52.923517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] The token cache is being invalidate because project 3bc739c34f634c1fb4f8cbd2186dbaa7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:39:52.923856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:52.924520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:52.931126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9177ec85-4c16-42d7-a9d0-e5202b884877 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:52.931651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1267/2534] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:39:52 2022] DELETE /identity/v3/projects/3bc739c34f634c1fb4f8cbd2186dbaa7 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:52.948184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9177ec85-4c16-42d7-a9d0-e5202b884877 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:52.976099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:53.038425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:53.039537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:53.040014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:53.040501 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] PATH_INFO: `/v3/users/1ace729a2c874bb2a18d1673163edf9d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:53.042695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ace729a2c874bb2a18d1673163edf9d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:53.045078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:53.108430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ace729a2c874bb2a18d1673163edf9d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:53.120537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ace729a2c874bb2a18d1673163edf9d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:53.122222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ace729a2c874bb2a18d1673163edf9d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:53.122898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ace729a2c874bb2a18d1673163edf9d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:53.136945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5fbe6e52-3e47-4a40-ad74-8e31c60ecfc7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ace729a2c874bb2a18d1673163edf9d'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:53.163274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1268/2535] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:39:52 2022] DELETE /identity/v3/users/1ace729a2c874bb2a18d1673163edf9d => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:53.673212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:53.675444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:53.697021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3919a0ae-25e7-45cf-b89f-535790195679 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:53.760476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:53.761413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:53.761737 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:53.762015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] PATH_INFO: `/v3/projects/02c11d71b8d540bb87a8261906076782` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:53.763197 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02c11d71b8d540bb87a8261906076782)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:53.765324 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:53.831645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02c11d71b8d540bb87a8261906076782', 'request_id': 'req-8fc41d43-0107-4c7b-9f46-1959c2528251'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:53.837712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02c11d71b8d540bb87a8261906076782', 'request_id': 'req-8fc41d43-0107-4c7b-9f46-1959c2528251'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:53.855534 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02c11d71b8d540bb87a8261906076782', 'request_id': 'req-8fc41d43-0107-4c7b-9f46-1959c2528251'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:53.878653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] The token cache is being invalidate because project 02c11d71b8d540bb87a8261906076782 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:39:53.878988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:53.879758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8fc41d43-0107-4c7b-9f46-1959c2528251 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:39:53.882625 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1268/2536] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:39:53 2022] DELETE /identity/v3/projects/02c11d71b8d540bb87a8261906076782 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:53.888263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d2e42-91a4-4e67-b222-622c15dfb35e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:53.888555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d2e42-91a4-4e67-b222-622c15dfb35e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:53.888826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810d2e42-91a4-4e67-b222-622c15dfb35e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:53.928814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-810d2e42-91a4-4e67-b222-622c15dfb35e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:53.930145 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-810d2e42-91a4-4e67-b222-622c15dfb35e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:54.027986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1269/2537] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:54.035878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dd12c8-6f80-4f31-a10f-9f36f9c1060d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.036168 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dd12c8-6f80-4f31-a10f-9f36f9c1060d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.036733 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dd12c8-6f80-4f31-a10f-9f36f9c1060d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.060157 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-a8dd12c8-6f80-4f31-a10f-9f36f9c1060d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:54.061430 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a8dd12c8-6f80-4f31-a10f-9f36f9c1060d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:54.071621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1269/2538] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:54.077113 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810d2e42-91a4-4e67-b222-622c15dfb35e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.090555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810d2e42-91a4-4e67-b222-622c15dfb35e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:54.095738 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704103b5-c69f-4ba6-b5ec-7ea4a60bae7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:54.097152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704103b5-c69f-4ba6-b5ec-7ea4a60bae7d admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.097574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704103b5-c69f-4ba6-b5ec-7ea4a60bae7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.098197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704103b5-c69f-4ba6-b5ec-7ea4a60bae7d admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.099123 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704103b5-c69f-4ba6-b5ec-7ea4a60bae7d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:39:54.099546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704103b5-c69f-4ba6-b5ec-7ea4a60bae7d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:54.101436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704103b5-c69f-4ba6-b5ec-7ea4a60bae7d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:54.130314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1270/2539] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:39:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:39:54.130984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dd12c8-6f80-4f31-a10f-9f36f9c1060d None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.141948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dd12c8-6f80-4f31-a10f-9f36f9c1060d None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:54.147107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b71070-0a3c-4504-a703-a55272026956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:54.147829 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b71070-0a3c-4504-a703-a55272026956 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.148199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b71070-0a3c-4504-a703-a55272026956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.148575 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b71070-0a3c-4504-a703-a55272026956 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.149300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b71070-0a3c-4504-a703-a55272026956 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:54.151440 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b71070-0a3c-4504-a703-a55272026956 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:54.210390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1270/2540] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:39:54 2022] POST /identity/v3/projects => generated 375 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:39:54.211620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704103b5-c69f-4ba6-b5ec-7ea4a60bae7d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.222398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704103b5-c69f-4ba6-b5ec-7ea4a60bae7d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:54.250128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:54.283954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfa3706-9b90-4b9b-8f49-839579813d8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:54.284564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa3706-9b90-4b9b-8f49-839579813d8a admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.284968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa3706-9b90-4b9b-8f49-839579813d8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.285241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa3706-9b90-4b9b-8f49-839579813d8a admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.285889 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfa3706-9b90-4b9b-8f49-839579813d8a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:54.287880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfa3706-9b90-4b9b-8f49-839579813d8a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:54.313564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1271/2541] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:39:54 2022] POST /identity/v3/users => generated 288 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:39:54.320935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b71070-0a3c-4504-a703-a55272026956 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.336210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b71070-0a3c-4504-a703-a55272026956 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:54.342646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b08f3a8-af69-4e7b-a3be-804bb04ebe36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:54.343259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b08f3a8-af69-4e7b-a3be-804bb04ebe36 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.343534 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b08f3a8-af69-4e7b-a3be-804bb04ebe36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.343800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b08f3a8-af69-4e7b-a3be-804bb04ebe36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.344462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b08f3a8-af69-4e7b-a3be-804bb04ebe36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:54.346146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b08f3a8-af69-4e7b-a3be-804bb04ebe36 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:54.356618 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1271/2542] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:54 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:54.360772 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfa3706-9b90-4b9b-8f49-839579813d8a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.371210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfa3706-9b90-4b9b-8f49-839579813d8a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:54.376419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a051d3f6-e287-41e5-ac48-6de4b1717c82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:54.377078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a051d3f6-e287-41e5-ac48-6de4b1717c82 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.377355 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a051d3f6-e287-41e5-ac48-6de4b1717c82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.377608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a051d3f6-e287-41e5-ac48-6de4b1717c82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.378287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a051d3f6-e287-41e5-ac48-6de4b1717c82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:54.379891 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a051d3f6-e287-41e5-ac48-6de4b1717c82 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:54.390489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1272/2543] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:54.394925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b08f3a8-af69-4e7b-a3be-804bb04ebe36 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.406522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b08f3a8-af69-4e7b-a3be-804bb04ebe36 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:54.411466 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c558aebb-5932-45b3-a83d-38426e76301a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:54.412128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c558aebb-5932-45b3-a83d-38426e76301a admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.412404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c558aebb-5932-45b3-a83d-38426e76301a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.412680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c558aebb-5932-45b3-a83d-38426e76301a admin admin] PATH_INFO: `/v3/projects/c2531b31853e48d6b71c2c215f8ec561/users/9718fd9007ed4a2381967282a976599b/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.427696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:DELETE=11,keystone:SELECT=86,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:54.431230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c558aebb-5932-45b3-a83d-38426e76301a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2531b31853e48d6b71c2c215f8ec561, user_id=9718fd9007ed4a2381967282a976599b, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:54.435812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c558aebb-5932-45b3-a83d-38426e76301a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:54.453322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1272/2544] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:39:54 2022] PUT /identity/v3/projects/c2531b31853e48d6b71c2c215f8ec561/users/9718fd9007ed4a2381967282a976599b/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:54.464093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5b447d-c039-43b5-8d8e-c3ac95df094d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.464662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5b447d-c039-43b5-8d8e-c3ac95df094d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.465021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5b447d-c039-43b5-8d8e-c3ac95df094d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.496013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4b5b447d-c039-43b5-8d8e-c3ac95df094d None None] MFA Rules not processed for user `9718fd9007ed4a2381967282a976599b`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:54.497634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4b5b447d-c039-43b5-8d8e-c3ac95df094d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:54.570222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1273/2545] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:54 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:39:54.575479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c558aebb-5932-45b3-a83d-38426e76301a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.652881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c558aebb-5932-45b3-a83d-38426e76301a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:54.663722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4279b218-3e90-4810-81ac-b4a54b9854e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:54.664389 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4279b218-3e90-4810-81ac-b4a54b9854e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.664670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4279b218-3e90-4810-81ac-b4a54b9854e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.664943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4279b218-3e90-4810-81ac-b4a54b9854e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.665589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4279b218-3e90-4810-81ac-b4a54b9854e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:54.668998 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4279b218-3e90-4810-81ac-b4a54b9854e2 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:54.704091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1273/2546] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:39:54 2022] POST /identity/v3/projects => generated 377 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:39:54.706484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5b447d-c039-43b5-8d8e-c3ac95df094d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.718894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5b447d-c039-43b5-8d8e-c3ac95df094d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:54.787884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf819bb1-aea2-46b8-8598-38d6075fdf97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:54.788939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf819bb1-aea2-46b8-8598-38d6075fdf97 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.789423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf819bb1-aea2-46b8-8598-38d6075fdf97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.789722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf819bb1-aea2-46b8-8598-38d6075fdf97 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.790415 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf819bb1-aea2-46b8-8598-38d6075fdf97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:54.792451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf819bb1-aea2-46b8-8598-38d6075fdf97 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:54.837599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1274/2547] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:39:54 2022] POST /identity/v3/users => generated 295 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:39:54.841138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4279b218-3e90-4810-81ac-b4a54b9854e2 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.852373 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4279b218-3e90-4810-81ac-b4a54b9854e2 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:54.858142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d532df8-5792-4bba-8af2-51a0c9c1f8a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:54.858777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d532df8-5792-4bba-8af2-51a0c9c1f8a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.859206 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d532df8-5792-4bba-8af2-51a0c9c1f8a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.859489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d532df8-5792-4bba-8af2-51a0c9c1f8a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.860139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d532df8-5792-4bba-8af2-51a0c9c1f8a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:54.861772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d532df8-5792-4bba-8af2-51a0c9c1f8a7 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:54.874658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1274/2548] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:54.879090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf819bb1-aea2-46b8-8598-38d6075fdf97 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.889697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf819bb1-aea2-46b8-8598-38d6075fdf97 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:54.894497 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ee0004-6d05-426c-baa4-8684832277ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:54.896458 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee0004-6d05-426c-baa4-8684832277ec admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:54.896743 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee0004-6d05-426c-baa4-8684832277ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:54.897127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee0004-6d05-426c-baa4-8684832277ec admin admin] PATH_INFO: `/v3/projects/9e397ef67f9d41519b9cd879db45b2e3/users/42da7f2125864f3284c45c981a7ef3eb/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:54.914159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ee0004-6d05-426c-baa4-8684832277ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e397ef67f9d41519b9cd879db45b2e3, user_id=42da7f2125864f3284c45c981a7ef3eb, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:54.917214 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ee0004-6d05-426c-baa4-8684832277ec admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:54.934796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1275/2549] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:39:54 2022] PUT /identity/v3/projects/9e397ef67f9d41519b9cd879db45b2e3/users/42da7f2125864f3284c45c981a7ef3eb/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:39:54.938725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d532df8-5792-4bba-8af2-51a0c9c1f8a7 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:54.950286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d532df8-5792-4bba-8af2-51a0c9c1f8a7 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:55.014183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19f5cfc-1d38-4143-b5cc-a7865d4259bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:55.015155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19f5cfc-1d38-4143-b5cc-a7865d4259bf admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:55.015433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19f5cfc-1d38-4143-b5cc-a7865d4259bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:55.015707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19f5cfc-1d38-4143-b5cc-a7865d4259bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:55.016502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19f5cfc-1d38-4143-b5cc-a7865d4259bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:55.018156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19f5cfc-1d38-4143-b5cc-a7865d4259bf admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:55.030639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1275/2550] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:39:54 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:39:55.034594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ee0004-6d05-426c-baa4-8684832277ec admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:55.040829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:55.048004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ee0004-6d05-426c-baa4-8684832277ec admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:55.063569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799904f4-9532-42c1-bb9b-8644fbcffd50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:55.064383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799904f4-9532-42c1-bb9b-8644fbcffd50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:55.069274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799904f4-9532-42c1-bb9b-8644fbcffd50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:55.069635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799904f4-9532-42c1-bb9b-8644fbcffd50 admin admin] PATH_INFO: `/v3/projects/9e397ef67f9d41519b9cd879db45b2e3/users/42da7f2125864f3284c45c981a7ef3eb/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:55.071917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799904f4-9532-42c1-bb9b-8644fbcffd50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e397ef67f9d41519b9cd879db45b2e3, user_id=42da7f2125864f3284c45c981a7ef3eb, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:55.075907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799904f4-9532-42c1-bb9b-8644fbcffd50 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:55.090478 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1276/2551] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:39:55 2022] PUT /identity/v3/projects/9e397ef67f9d41519b9cd879db45b2e3/users/42da7f2125864f3284c45c981a7ef3eb/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:39:55.101519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b9ce61-2f80-45a8-af72-b9ba8b0234b6 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:55.101799 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b9ce61-2f80-45a8-af72-b9ba8b0234b6 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:55.102066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b9ce61-2f80-45a8-af72-b9ba8b0234b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:55.133260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-17b9ce61-2f80-45a8-af72-b9ba8b0234b6 None None] MFA Rules not processed for user `42da7f2125864f3284c45c981a7ef3eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:39:55.134771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-17b9ce61-2f80-45a8-af72-b9ba8b0234b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:55.194877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:39:55.227689 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1276/2552] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:39:55 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:39:55.241034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799904f4-9532-42c1-bb9b-8644fbcffd50 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:55.243059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-799904f4-9532-42c1-bb9b-8644fbcffd50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:55.311839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799904f4-9532-42c1-bb9b-8644fbcffd50 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:55.315705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826b647a-c114-47a8-8d7f-75b39f7ce8b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:55.316321 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826b647a-c114-47a8-8d7f-75b39f7ce8b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:55.316602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826b647a-c114-47a8-8d7f-75b39f7ce8b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:55.316871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826b647a-c114-47a8-8d7f-75b39f7ce8b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:55.380987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826b647a-c114-47a8-8d7f-75b39f7ce8b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:55.382712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826b647a-c114-47a8-8d7f-75b39f7ce8b0 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:55.392384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1277/2553] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:55 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:39:57.777449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b9ce61-2f80-45a8-af72-b9ba8b0234b6 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:39:57.779094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-17b9ce61-2f80-45a8-af72-b9ba8b0234b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:57.849565 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b9ce61-2f80-45a8-af72-b9ba8b0234b6 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:39:57.853891 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546e70fe-ce24-4e44-9a80-7e9654c66f0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:39:57.854535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546e70fe-ce24-4e44-9a80-7e9654c66f0a service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:39:57.854810 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546e70fe-ce24-4e44-9a80-7e9654c66f0a service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:39:57.855079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546e70fe-ce24-4e44-9a80-7e9654c66f0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:39:57.857445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-546e70fe-ce24-4e44-9a80-7e9654c66f0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:39:57.924300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546e70fe-ce24-4e44-9a80-7e9654c66f0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:39:57.925751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546e70fe-ce24-4e44-9a80-7e9654c66f0a service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:39:57.936246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1277/2554] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:39:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:01.419780 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826b647a-c114-47a8-8d7f-75b39f7ce8b0 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:01.421453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-826b647a-c114-47a8-8d7f-75b39f7ce8b0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:01.436111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826b647a-c114-47a8-8d7f-75b39f7ce8b0 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:01.441672 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:01.442309 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:01.442589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:01.442867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] PATH_INFO: `/v3/users/515e0f8a3a234deba5f909271b303abb` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:01.458084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=515e0f8a3a234deba5f909271b303abb)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:01.460042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:01.513694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '515e0f8a3a234deba5f909271b303abb'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:01.523361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '515e0f8a3a234deba5f909271b303abb'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:01.524047 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '515e0f8a3a234deba5f909271b303abb'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:01.524844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '515e0f8a3a234deba5f909271b303abb'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:01.532086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-35478755-87d4-4265-8887-c40752b28fd8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '515e0f8a3a234deba5f909271b303abb'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:01.549750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1278/2555] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:01 2022] DELETE /identity/v3/users/515e0f8a3a234deba5f909271b303abb => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:01.730988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546e70fe-ce24-4e44-9a80-7e9654c66f0a service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:01.742905 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546e70fe-ce24-4e44-9a80-7e9654c66f0a service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:01.804147 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:01.804768 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:01.805048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:01.805318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] PATH_INFO: `/v3/projects/3cbf0c034dd54a8d856d9bc6fb485b45` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:01.816187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3cbf0c034dd54a8d856d9bc6fb485b45)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:01.818187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:01.856344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:01.871902 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3cbf0c034dd54a8d856d9bc6fb485b45', 'request_id': 'req-7f7072b0-0400-46cd-a9ba-66bed243f241'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:01.877870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3cbf0c034dd54a8d856d9bc6fb485b45', 'request_id': 'req-7f7072b0-0400-46cd-a9ba-66bed243f241'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:01.886942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3cbf0c034dd54a8d856d9bc6fb485b45', 'request_id': 'req-7f7072b0-0400-46cd-a9ba-66bed243f241'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:01.898699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] The token cache is being invalidate because project 3cbf0c034dd54a8d856d9bc6fb485b45 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:01.898987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:01.899634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7f7072b0-0400-46cd-a9ba-66bed243f241 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:01.901132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1278/2556] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:01 2022] DELETE /identity/v3/projects/3cbf0c034dd54a8d856d9bc6fb485b45 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:01.906594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c37ae7-e754-4a65-b83a-60faf26e9be5 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:01.906883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c37ae7-e754-4a65-b83a-60faf26e9be5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:01.907176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c37ae7-e754-4a65-b83a-60faf26e9be5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:01.916280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:01.927957 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-97c37ae7-e754-4a65-b83a-60faf26e9be5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:01.929750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-97c37ae7-e754-4a65-b83a-60faf26e9be5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:01.993241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1279/2557] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:02.001567 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e7615f-cde8-49b2-81aa-64a6ee18fc4b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.001855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e7615f-cde8-49b2-81aa-64a6ee18fc4b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.002140 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e7615f-cde8-49b2-81aa-64a6ee18fc4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.022658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d4e7615f-cde8-49b2-81aa-64a6ee18fc4b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:02.023894 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d4e7615f-cde8-49b2-81aa-64a6ee18fc4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:02.032917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1279/2558] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:02.037254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c37ae7-e754-4a65-b83a-60faf26e9be5 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:02.047522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c37ae7-e754-4a65-b83a-60faf26e9be5 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:02.052464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c76e18f-8fbc-4946-9aa0-40987a02edcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:02.053071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c76e18f-8fbc-4946-9aa0-40987a02edcd admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.053349 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c76e18f-8fbc-4946-9aa0-40987a02edcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.053621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c76e18f-8fbc-4946-9aa0-40987a02edcd admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.054300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c76e18f-8fbc-4946-9aa0-40987a02edcd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:40:02.054601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c76e18f-8fbc-4946-9aa0-40987a02edcd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:02.056249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c76e18f-8fbc-4946-9aa0-40987a02edcd admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:02.067811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1280/2559] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:40:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:40:02.072605 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e7615f-cde8-49b2-81aa-64a6ee18fc4b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:02.082404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e7615f-cde8-49b2-81aa-64a6ee18fc4b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:02.087999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e42e0c-1349-462c-a5aa-480f25554df4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:02.088639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e42e0c-1349-462c-a5aa-480f25554df4 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.088921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e42e0c-1349-462c-a5aa-480f25554df4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.089193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e42e0c-1349-462c-a5aa-480f25554df4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.090177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e42e0c-1349-462c-a5aa-480f25554df4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:02.091782 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e42e0c-1349-462c-a5aa-480f25554df4 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:02.131502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1280/2560] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:02 2022] POST /identity/v3/projects => generated 383 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:02.132514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e42e0c-1349-462c-a5aa-480f25554df4 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:02.132653 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c76e18f-8fbc-4946-9aa0-40987a02edcd admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:02.134757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2c76e18f-8fbc-4946-9aa0-40987a02edcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:02.197339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e42e0c-1349-462c-a5aa-480f25554df4 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:02.202068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11315919-0a4d-4d3f-b2b2-1bb644a43911 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:02.202713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11315919-0a4d-4d3f-b2b2-1bb644a43911 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.202991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11315919-0a4d-4d3f-b2b2-1bb644a43911 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.203260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11315919-0a4d-4d3f-b2b2-1bb644a43911 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.203895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11315919-0a4d-4d3f-b2b2-1bb644a43911 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:02.205835 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11315919-0a4d-4d3f-b2b2-1bb644a43911 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:02.206461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c76e18f-8fbc-4946-9aa0-40987a02edcd admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:02.229100 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1281/2561] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:02 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:02.234063 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11315919-0a4d-4d3f-b2b2-1bb644a43911 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:02.243630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11315919-0a4d-4d3f-b2b2-1bb644a43911 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:02.248630 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837d00c-39f7-4f97-a5e9-d1ba56dd9b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:02.249218 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837d00c-39f7-4f97-a5e9-d1ba56dd9b46 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.249498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837d00c-39f7-4f97-a5e9-d1ba56dd9b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.249770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6837d00c-39f7-4f97-a5e9-d1ba56dd9b46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.250442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6837d00c-39f7-4f97-a5e9-d1ba56dd9b46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:02.252048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6837d00c-39f7-4f97-a5e9-d1ba56dd9b46 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:02.258505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b85c75-6f65-47f4-85a9-e1519ffae013 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:02.259146 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b85c75-6f65-47f4-85a9-e1519ffae013 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.259394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b85c75-6f65-47f4-85a9-e1519ffae013 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.259633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b85c75-6f65-47f4-85a9-e1519ffae013 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.261472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-41b85c75-6f65-47f4-85a9-e1519ffae013 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:02.261877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1282/2562] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:02 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:02.267041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837d00c-39f7-4f97-a5e9-d1ba56dd9b46 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:02.276633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6837d00c-39f7-4f97-a5e9-d1ba56dd9b46 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:02.281423 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82712685-85ad-4c78-8ece-fcdc7ed4fded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:02.282008 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82712685-85ad-4c78-8ece-fcdc7ed4fded admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.282326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82712685-85ad-4c78-8ece-fcdc7ed4fded admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.282702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82712685-85ad-4c78-8ece-fcdc7ed4fded admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.283358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82712685-85ad-4c78-8ece-fcdc7ed4fded admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:02.284953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82712685-85ad-4c78-8ece-fcdc7ed4fded admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:02.296350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1283/2563] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:02.300374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82712685-85ad-4c78-8ece-fcdc7ed4fded admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:02.310021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82712685-85ad-4c78-8ece-fcdc7ed4fded admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:02.315339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b583f3-b986-429b-80c2-ebad56a9730f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:02.316132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b583f3-b986-429b-80c2-ebad56a9730f admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.316415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b583f3-b986-429b-80c2-ebad56a9730f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.316680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b583f3-b986-429b-80c2-ebad56a9730f admin admin] PATH_INFO: `/v3/projects/bf9c6d4a95e549b1a1931e613b834afc/users/df66e899b9594487bef79b2fbc079bb8/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.329158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b85c75-6f65-47f4-85a9-e1519ffae013 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:02.330708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b85c75-6f65-47f4-85a9-e1519ffae013 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:02.332422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b583f3-b986-429b-80c2-ebad56a9730f admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf9c6d4a95e549b1a1931e613b834afc, user_id=df66e899b9594487bef79b2fbc079bb8, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:02.334490 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:02.335106 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b583f3-b986-429b-80c2-ebad56a9730f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:02.342390 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1281/2564] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:02 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:02.349836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1284/2565] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:02 2022] PUT /identity/v3/projects/bf9c6d4a95e549b1a1931e613b834afc/users/df66e899b9594487bef79b2fbc079bb8/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:40:02.360322 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87c1539-c568-4f92-a2dd-697a65569135 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.360593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87c1539-c568-4f92-a2dd-697a65569135 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.360855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87c1539-c568-4f92-a2dd-697a65569135 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.388694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b87c1539-c568-4f92-a2dd-697a65569135 None None] MFA Rules not processed for user `df66e899b9594487bef79b2fbc079bb8`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:02.390315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b87c1539-c568-4f92-a2dd-697a65569135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:02.457821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1282/2566] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:02 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:02.469885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b583f3-b986-429b-80c2-ebad56a9730f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:02.472190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-00b583f3-b986-429b-80c2-ebad56a9730f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:02.509259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=5,keystone:INSERT=6 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:02.553579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b583f3-b986-429b-80c2-ebad56a9730f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:02.559171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff815e0-4d12-44a4-ae7a-0f42d1dcc0a2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:02.559794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff815e0-4d12-44a4-ae7a-0f42d1dcc0a2 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.560307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff815e0-4d12-44a4-ae7a-0f42d1dcc0a2 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.560307 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff815e0-4d12-44a4-ae7a-0f42d1dcc0a2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.568843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff815e0-4d12-44a4-ae7a-0f42d1dcc0a2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:02.570496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff815e0-4d12-44a4-ae7a-0f42d1dcc0a2 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:02.579988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1285/2567] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:02.602610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87c1539-c568-4f92-a2dd-697a65569135 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:02.604043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b87c1539-c568-4f92-a2dd-697a65569135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:02.677880 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87c1539-c568-4f92-a2dd-697a65569135 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:02.683188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefe12b0-cbde-4277-b477-3551bb1610ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:02.684071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefe12b0-cbde-4277-b477-3551bb1610ff service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:02.684450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefe12b0-cbde-4277-b477-3551bb1610ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:02.684818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefe12b0-cbde-4277-b477-3551bb1610ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:02.686955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-aefe12b0-cbde-4277-b477-3551bb1610ff service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:02.703334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aefe12b0-cbde-4277-b477-3551bb1610ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:02.704850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aefe12b0-cbde-4277-b477-3551bb1610ff service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:02.716840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1283/2568] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:12.576178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=19 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:12.712105 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=59 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:20.668398 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff815e0-4d12-44a4-ae7a-0f42d1dcc0a2 service nova] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:20.730473 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff815e0-4d12-44a4-ae7a-0f42d1dcc0a2 service nova] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:20.735871 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:20.736494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:20.736769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:20.737035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] PATH_INFO: `/v3/users/df66e899b9594487bef79b2fbc079bb8` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:20.738195 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] RBAC: Authorizing `identity:delete_user(user_id=df66e899b9594487bef79b2fbc079bb8)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:20.740106 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:20.789962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df66e899b9594487bef79b2fbc079bb8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:20.790642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df66e899b9594487bef79b2fbc079bb8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:20.797695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df66e899b9594487bef79b2fbc079bb8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:20.798322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df66e899b9594487bef79b2fbc079bb8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:20.806926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df66e899b9594487bef79b2fbc079bb8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:20.825192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1286/2569] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:20 2022] DELETE /identity/v3/users/df66e899b9594487bef79b2fbc079bb8 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:40:20.830082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6f2f75-f243-4cbc-9f18-689aba9f7b81 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:20.830586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6f2f75-f243-4cbc-9f18-689aba9f7b81 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:20.830726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6f2f75-f243-4cbc-9f18-689aba9f7b81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:20.851476 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0c6f2f75-f243-4cbc-9f18-689aba9f7b81 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:20.852722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0c6f2f75-f243-4cbc-9f18-689aba9f7b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:20.915205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1284/2570] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:20.925449 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:20.927094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:20.990090 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38780399-3610-4233-b926-ff44fe4b3c2a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:20.994550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400c0de0-575e-40fd-ab7c-3e4714386c14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:20.995149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400c0de0-575e-40fd-ab7c-3e4714386c14 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:20.995426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400c0de0-575e-40fd-ab7c-3e4714386c14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:20.995690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400c0de0-575e-40fd-ab7c-3e4714386c14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.005018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400c0de0-575e-40fd-ab7c-3e4714386c14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:21.006510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400c0de0-575e-40fd-ab7c-3e4714386c14 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:21.015822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1287/2571] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:21.316618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6f2f75-f243-4cbc-9f18-689aba9f7b81 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:21.318166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0c6f2f75-f243-4cbc-9f18-689aba9f7b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:21.331064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6f2f75-f243-4cbc-9f18-689aba9f7b81 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:21.336630 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:21.337417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:21.337804 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:21.338211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] PATH_INFO: `/v3/projects/bf9c6d4a95e549b1a1931e613b834afc` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.339781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf9c6d4a95e549b1a1931e613b834afc)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:21.341760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:21.392062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf9c6d4a95e549b1a1931e613b834afc', 'request_id': 'req-e24e722c-70f5-4aa9-8ee5-c902ee64391a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:21.397517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf9c6d4a95e549b1a1931e613b834afc', 'request_id': 'req-e24e722c-70f5-4aa9-8ee5-c902ee64391a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:21.406443 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf9c6d4a95e549b1a1931e613b834afc', 'request_id': 'req-e24e722c-70f5-4aa9-8ee5-c902ee64391a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:21.417588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] The token cache is being invalidate because project bf9c6d4a95e549b1a1931e613b834afc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:21.417875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:21.418589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e24e722c-70f5-4aa9-8ee5-c902ee64391a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:21.420129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1285/2572] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:21 2022] DELETE /identity/v3/projects/bf9c6d4a95e549b1a1931e613b834afc => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:21.425263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee74b0-6680-42fa-b4b3-86ab220f7e3f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:21.425545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee74b0-6680-42fa-b4b3-86ab220f7e3f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:21.425816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee74b0-6680-42fa-b4b3-86ab220f7e3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.445753 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-22ee74b0-6680-42fa-b4b3-86ab220f7e3f None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:21.446999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-22ee74b0-6680-42fa-b4b3-86ab220f7e3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:21.465736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:21.515239 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1288/2573] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:21.524676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fc4dd6-c775-4b99-b090-279a9d4c6fa5 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:21.524967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fc4dd6-c775-4b99-b090-279a9d4c6fa5 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:21.525226 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fc4dd6-c775-4b99-b090-279a9d4c6fa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.546562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-08fc4dd6-c775-4b99-b090-279a9d4c6fa5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:21.547778 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-08fc4dd6-c775-4b99-b090-279a9d4c6fa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:21.556370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1286/2574] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:21.562825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee74b0-6680-42fa-b4b3-86ab220f7e3f None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:21.575484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee74b0-6680-42fa-b4b3-86ab220f7e3f None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:21.582535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68dc190-537f-4bde-bdd6-aac32507de1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:21.583142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68dc190-537f-4bde-bdd6-aac32507de1e admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:21.583420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68dc190-537f-4bde-bdd6-aac32507de1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:21.583660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68dc190-537f-4bde-bdd6-aac32507de1e admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.584304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68dc190-537f-4bde-bdd6-aac32507de1e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:40:21.584602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68dc190-537f-4bde-bdd6-aac32507de1e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:21.586235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68dc190-537f-4bde-bdd6-aac32507de1e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:21.600003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1289/2575] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:40:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:40:21.614646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fc4dd6-c775-4b99-b090-279a9d4c6fa5 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:21.627120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fc4dd6-c775-4b99-b090-279a9d4c6fa5 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:21.636060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae58c1a-d735-4a48-80ed-9f19a6ec9a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:21.637027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae58c1a-d735-4a48-80ed-9f19a6ec9a57 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:21.637457 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae58c1a-d735-4a48-80ed-9f19a6ec9a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:21.637886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae58c1a-d735-4a48-80ed-9f19a6ec9a57 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.638970 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae58c1a-d735-4a48-80ed-9f19a6ec9a57 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:21.641335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae58c1a-d735-4a48-80ed-9f19a6ec9a57 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:21.671709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1287/2576] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:21 2022] POST /identity/v3/projects => generated 367 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:21.676185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68dc190-537f-4bde-bdd6-aac32507de1e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:21.687274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68dc190-537f-4bde-bdd6-aac32507de1e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:21.751411 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d333211d-c62f-4234-9ff9-2618bc914609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:21.752024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d333211d-c62f-4234-9ff9-2618bc914609 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:21.752281 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d333211d-c62f-4234-9ff9-2618bc914609 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:21.752549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d333211d-c62f-4234-9ff9-2618bc914609 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.753187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d333211d-c62f-4234-9ff9-2618bc914609 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:21.755146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d333211d-c62f-4234-9ff9-2618bc914609 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:21.777465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1290/2577] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:21 2022] POST /identity/v3/users => generated 284 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:40:21.781793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae58c1a-d735-4a48-80ed-9f19a6ec9a57 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:21.792382 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae58c1a-d735-4a48-80ed-9f19a6ec9a57 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:21.798091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1778c92d-5eaa-4dd7-a294-0419966f26c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:21.798722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1778c92d-5eaa-4dd7-a294-0419966f26c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:21.798993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1778c92d-5eaa-4dd7-a294-0419966f26c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:21.799256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1778c92d-5eaa-4dd7-a294-0419966f26c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.799895 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1778c92d-5eaa-4dd7-a294-0419966f26c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:21.801514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1778c92d-5eaa-4dd7-a294-0419966f26c6 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:21.812452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1288/2578] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:21.816509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d333211d-c62f-4234-9ff9-2618bc914609 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:21.827171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d333211d-c62f-4234-9ff9-2618bc914609 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:21.832104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edc2acf-dccb-4b54-bff2-838cdba1ce53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:21.832697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edc2acf-dccb-4b54-bff2-838cdba1ce53 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:21.832975 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edc2acf-dccb-4b54-bff2-838cdba1ce53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:21.833238 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edc2acf-dccb-4b54-bff2-838cdba1ce53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.833865 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edc2acf-dccb-4b54-bff2-838cdba1ce53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:21.835504 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edc2acf-dccb-4b54-bff2-838cdba1ce53 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:21.846139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1291/2579] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:21.850394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1778c92d-5eaa-4dd7-a294-0419966f26c6 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:21.861155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1778c92d-5eaa-4dd7-a294-0419966f26c6 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:21.866465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8694486-8f1b-483b-aff8-baaeeccec23d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:21.867046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8694486-8f1b-483b-aff8-baaeeccec23d admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:21.867320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8694486-8f1b-483b-aff8-baaeeccec23d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:21.867583 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8694486-8f1b-483b-aff8-baaeeccec23d admin admin] PATH_INFO: `/v3/projects/ff00969c7f524d60beb88f1d85f18073/users/1c7f6ee6583a4c0e86c6bec9135f0816/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.883261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8694486-8f1b-483b-aff8-baaeeccec23d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff00969c7f524d60beb88f1d85f18073, user_id=1c7f6ee6583a4c0e86c6bec9135f0816, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:21.885883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8694486-8f1b-483b-aff8-baaeeccec23d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:21.900926 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1289/2580] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:21 2022] PUT /identity/v3/projects/ff00969c7f524d60beb88f1d85f18073/users/1c7f6ee6583a4c0e86c6bec9135f0816/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:21.911379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e812077c-9665-409a-8a17-d7a52947c2b7 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:21.911663 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e812077c-9665-409a-8a17-d7a52947c2b7 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:21.911935 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e812077c-9665-409a-8a17-d7a52947c2b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:21.939861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e812077c-9665-409a-8a17-d7a52947c2b7 None None] MFA Rules not processed for user `1c7f6ee6583a4c0e86c6bec9135f0816`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:21.941358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e812077c-9665-409a-8a17-d7a52947c2b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:22.008369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1292/2581] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:21 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:22.024726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8694486-8f1b-483b-aff8-baaeeccec23d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:22.026783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b8694486-8f1b-483b-aff8-baaeeccec23d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:22.050894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:22.089938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8694486-8f1b-483b-aff8-baaeeccec23d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:22.094154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d78f7d-e5da-4724-abce-b3c880c72a14 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:22.094815 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d78f7d-e5da-4724-abce-b3c880c72a14 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:22.095100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d78f7d-e5da-4724-abce-b3c880c72a14 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:22.095364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d78f7d-e5da-4724-abce-b3c880c72a14 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:22.104642 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d78f7d-e5da-4724-abce-b3c880c72a14 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:22.106064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d78f7d-e5da-4724-abce-b3c880c72a14 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:22.115897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1290/2582] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:22 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:32.003127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:32.111572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=17 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:35.791518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e812077c-9665-409a-8a17-d7a52947c2b7 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:35.793209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e812077c-9665-409a-8a17-d7a52947c2b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:35.869300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e812077c-9665-409a-8a17-d7a52947c2b7 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:35.874168 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:35.874821 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:35.875082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:35.875356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] PATH_INFO: `/v3/users/9718fd9007ed4a2381967282a976599b` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:35.876528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9718fd9007ed4a2381967282a976599b)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:35.878498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:35.925730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9718fd9007ed4a2381967282a976599b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:35.926379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9718fd9007ed4a2381967282a976599b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:35.933051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9718fd9007ed4a2381967282a976599b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:35.933661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9718fd9007ed4a2381967282a976599b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:35.942571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9718fd9007ed4a2381967282a976599b'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:35.961075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1293/2583] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:35 2022] DELETE /identity/v3/users/9718fd9007ed4a2381967282a976599b => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:35.965694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b032f467-05bd-44db-858e-3b734b55fb4b None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:35.966240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b032f467-05bd-44db-858e-3b734b55fb4b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:35.966240 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b032f467-05bd-44db-858e-3b734b55fb4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:35.985351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-b032f467-05bd-44db-858e-3b734b55fb4b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:35.986585 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b032f467-05bd-44db-858e-3b734b55fb4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:36.050729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1291/2584] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:36.061067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:36.062736 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:36.123567 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5134e8-f39d-4cbe-aaf6-0e891b30e0f5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:36.127764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60d5299-832f-4bc7-9d2e-d0c5e78c3e58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:36.128369 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60d5299-832f-4bc7-9d2e-d0c5e78c3e58 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:36.128639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60d5299-832f-4bc7-9d2e-d0c5e78c3e58 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:36.128903 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60d5299-832f-4bc7-9d2e-d0c5e78c3e58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:36.141178 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60d5299-832f-4bc7-9d2e-d0c5e78c3e58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:36.142769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60d5299-832f-4bc7-9d2e-d0c5e78c3e58 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:36.153527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1294/2585] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:36.440100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b032f467-05bd-44db-858e-3b734b55fb4b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:36.441600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b032f467-05bd-44db-858e-3b734b55fb4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:36.455038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b032f467-05bd-44db-858e-3b734b55fb4b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:36.460803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:36.461451 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:36.461725 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:36.461985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] PATH_INFO: `/v3/projects/c2531b31853e48d6b71c2c215f8ec561` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:36.463373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c2531b31853e48d6b71c2c215f8ec561)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:36.465577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:36.513670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c2531b31853e48d6b71c2c215f8ec561', 'request_id': 'req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:36.519093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c2531b31853e48d6b71c2c215f8ec561', 'request_id': 'req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:36.527712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c2531b31853e48d6b71c2c215f8ec561', 'request_id': 'req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:36.539064 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] The token cache is being invalidate because project c2531b31853e48d6b71c2c215f8ec561 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:36.539351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:36.539968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:36.541646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1292/2586] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:36 2022] DELETE /identity/v3/projects/c2531b31853e48d6b71c2c215f8ec561 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:36.546912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60d5299-832f-4bc7-9d2e-d0c5e78c3e58 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:36.556548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60d5299-832f-4bc7-9d2e-d0c5e78c3e58 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:36.576009 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:36.624516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:36.625138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:36.625417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:36.625693 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] PATH_INFO: `/v3/users/42da7f2125864f3284c45c981a7ef3eb` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:36.627057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=42da7f2125864f3284c45c981a7ef3eb)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:36.629541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:36.698823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42da7f2125864f3284c45c981a7ef3eb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:36.699495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42da7f2125864f3284c45c981a7ef3eb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:36.706943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42da7f2125864f3284c45c981a7ef3eb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:36.707602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42da7f2125864f3284c45c981a7ef3eb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:36.717485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f5ee351e-2efb-4b26-991b-ce88509256ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42da7f2125864f3284c45c981a7ef3eb'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:36.739600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1295/2587] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:36 2022] DELETE /identity/v3/users/42da7f2125864f3284c45c981a7ef3eb => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:37.030129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.030129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:37.040905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc472a05-9571-4bb9-90c1-8d7c3b4346f8 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.101190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.101828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.102125 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.102410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] PATH_INFO: `/v3/projects/9e397ef67f9d41519b9cd879db45b2e3` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.103530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e397ef67f9d41519b9cd879db45b2e3)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.105468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.138535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:37.158989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e397ef67f9d41519b9cd879db45b2e3', 'request_id': 'req-186cff27-cd8c-4af2-8a0a-91791bc0b449'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:37.164590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e397ef67f9d41519b9cd879db45b2e3', 'request_id': 'req-186cff27-cd8c-4af2-8a0a-91791bc0b449'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:37.173115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e397ef67f9d41519b9cd879db45b2e3', 'request_id': 'req-186cff27-cd8c-4af2-8a0a-91791bc0b449'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:37.184985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] The token cache is being invalidate because project 9e397ef67f9d41519b9cd879db45b2e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:37.185267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:37.185933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-186cff27-cd8c-4af2-8a0a-91791bc0b449 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:37.187493 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1293/2588] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:37 2022] DELETE /identity/v3/projects/9e397ef67f9d41519b9cd879db45b2e3 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:37.192592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ea1fc0-8955-4aa6-a767-88caebb5d703 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.193098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ea1fc0-8955-4aa6-a767-88caebb5d703 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.193098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ea1fc0-8955-4aa6-a767-88caebb5d703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.227232 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-73ea1fc0-8955-4aa6-a767-88caebb5d703 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:37.228457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-73ea1fc0-8955-4aa6-a767-88caebb5d703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:37.302764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1296/2589] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:37.311448 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e28382c-7060-4160-967c-a2b3096b5dfc None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.311726 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e28382c-7060-4160-967c-a2b3096b5dfc None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.311998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e28382c-7060-4160-967c-a2b3096b5dfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.334945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-2e28382c-7060-4160-967c-a2b3096b5dfc None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:37.335936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ea1fc0-8955-4aa6-a767-88caebb5d703 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.336202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2e28382c-7060-4160-967c-a2b3096b5dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:37.337356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-73ea1fc0-8955-4aa6-a767-88caebb5d703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:37.345159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1294/2590] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:37.349286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e28382c-7060-4160-967c-a2b3096b5dfc None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.349563 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ea1fc0-8955-4aa6-a767-88caebb5d703 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.354920 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.355588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.355861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.356129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] PATH_INFO: `/v3/users/1c7f6ee6583a4c0e86c6bec9135f0816` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.357270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c7f6ee6583a4c0e86c6bec9135f0816)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.359109 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e28382c-7060-4160-967c-a2b3096b5dfc None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.359329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.363801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbc64fd-852e-4ac6-b48a-2312ad2a2406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.364394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbc64fd-852e-4ac6-b48a-2312ad2a2406 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.364665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbc64fd-852e-4ac6-b48a-2312ad2a2406 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.364918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbc64fd-852e-4ac6-b48a-2312ad2a2406 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.365553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbc64fd-852e-4ac6-b48a-2312ad2a2406 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:40:37.365860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbc64fd-852e-4ac6-b48a-2312ad2a2406 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.367466 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbc64fd-852e-4ac6-b48a-2312ad2a2406 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.381227 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1295/2591] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:40:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:40:37.385947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbc64fd-852e-4ac6-b48a-2312ad2a2406 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.396395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbc64fd-852e-4ac6-b48a-2312ad2a2406 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.401139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a43a6e-9286-4626-97ad-cc7e4815bc66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.401611 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a43a6e-9286-4626-97ad-cc7e4815bc66 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.401873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a43a6e-9286-4626-97ad-cc7e4815bc66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.402159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a43a6e-9286-4626-97ad-cc7e4815bc66 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.402797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a43a6e-9286-4626-97ad-cc7e4815bc66 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.402957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:37.404724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a43a6e-9286-4626-97ad-cc7e4815bc66 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.413817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c7f6ee6583a4c0e86c6bec9135f0816'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:37.414541 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c7f6ee6583a4c0e86c6bec9135f0816'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:37.422377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c7f6ee6583a4c0e86c6bec9135f0816'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:37.423031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c7f6ee6583a4c0e86c6bec9135f0816'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:37.430758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a977a21f-6df2-4ae8-8e68-0ac4f7799269 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c7f6ee6583a4c0e86c6bec9135f0816'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:37.435535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1296/2592] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:37 2022] POST /identity/v3/projects => generated 369 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:40:37.439608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a43a6e-9286-4626-97ad-cc7e4815bc66 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.441380 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-90a43a6e-9286-4626-97ad-cc7e4815bc66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:37.451954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1297/2593] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:37 2022] DELETE /identity/v3/users/1c7f6ee6583a4c0e86c6bec9135f0816 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:37.456421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bf9877-f6ed-45fa-865a-fa3114def0f7 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.456706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bf9877-f6ed-45fa-865a-fa3114def0f7 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.456972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bf9877-f6ed-45fa-865a-fa3114def0f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.477322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-72bf9877-f6ed-45fa-865a-fa3114def0f7 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:37.478548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-72bf9877-f6ed-45fa-865a-fa3114def0f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:37.512037 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a43a6e-9286-4626-97ad-cc7e4815bc66 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.540248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1298/2594] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:37.550796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bf9877-f6ed-45fa-865a-fa3114def0f7 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.552273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-72bf9877-f6ed-45fa-865a-fa3114def0f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:37.569777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681df8ec-76b9-43c8-b205-73ad1f162839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.570421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681df8ec-76b9-43c8-b205-73ad1f162839 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.570691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681df8ec-76b9-43c8-b205-73ad1f162839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.570955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681df8ec-76b9-43c8-b205-73ad1f162839 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.571588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681df8ec-76b9-43c8-b205-73ad1f162839 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.573549 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681df8ec-76b9-43c8-b205-73ad1f162839 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.597477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1297/2595] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:37 2022] POST /identity/v3/users => generated 285 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:37.601117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681df8ec-76b9-43c8-b205-73ad1f162839 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.611139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681df8ec-76b9-43c8-b205-73ad1f162839 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.611763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bf9877-f6ed-45fa-865a-fa3114def0f7 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.615339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9900be9-74ae-4f7b-9d4b-82385eaf8aba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.615954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9900be9-74ae-4f7b-9d4b-82385eaf8aba service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.616228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9900be9-74ae-4f7b-9d4b-82385eaf8aba service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.616499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9900be9-74ae-4f7b-9d4b-82385eaf8aba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.621264 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd2f095-42d6-4b29-8b08-a599a9512ddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.621881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd2f095-42d6-4b29-8b08-a599a9512ddc admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.622184 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd2f095-42d6-4b29-8b08-a599a9512ddc admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.622454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd2f095-42d6-4b29-8b08-a599a9512ddc admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.623101 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd2f095-42d6-4b29-8b08-a599a9512ddc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.624709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd2f095-42d6-4b29-8b08-a599a9512ddc admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.627073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9900be9-74ae-4f7b-9d4b-82385eaf8aba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.628549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9900be9-74ae-4f7b-9d4b-82385eaf8aba service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.635548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=6,keystone:SELECT=89,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:37.636001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1298/2596] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:37 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:37.639221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1299/2597] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:37.642843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd2f095-42d6-4b29-8b08-a599a9512ddc admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.656036 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd2f095-42d6-4b29-8b08-a599a9512ddc admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.665076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27bc72b-584a-44bf-bdaa-574329d04e67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.665631 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27bc72b-584a-44bf-bdaa-574329d04e67 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.665908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27bc72b-584a-44bf-bdaa-574329d04e67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.666210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27bc72b-584a-44bf-bdaa-574329d04e67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.666992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27bc72b-584a-44bf-bdaa-574329d04e67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.668620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27bc72b-584a-44bf-bdaa-574329d04e67 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.680015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1299/2598] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:37 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:37.684823 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9900be9-74ae-4f7b-9d4b-82385eaf8aba service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.695172 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9900be9-74ae-4f7b-9d4b-82385eaf8aba service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.701214 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d8fc43-7f59-4ad2-8390-49ec599a71b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.701876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d8fc43-7f59-4ad2-8390-49ec599a71b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.702181 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d8fc43-7f59-4ad2-8390-49ec599a71b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.702456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d8fc43-7f59-4ad2-8390-49ec599a71b3 admin admin] PATH_INFO: `/v3/projects/29252a7765dc4332880995290f130f9d/users/a7c77c879fda42a5907b53c36dfb9dab/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.719001 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d8fc43-7f59-4ad2-8390-49ec599a71b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=29252a7765dc4332880995290f130f9d, user_id=a7c77c879fda42a5907b53c36dfb9dab, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.721649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d8fc43-7f59-4ad2-8390-49ec599a71b3 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.736082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1300/2599] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:37 2022] PUT /identity/v3/projects/29252a7765dc4332880995290f130f9d/users/a7c77c879fda42a5907b53c36dfb9dab/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:40:37.748950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683989b1-8964-4474-8b4e-c31ced0c2020 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.749235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683989b1-8964-4474-8b4e-c31ced0c2020 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.749504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683989b1-8964-4474-8b4e-c31ced0c2020 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.778086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-683989b1-8964-4474-8b4e-c31ced0c2020 None None] MFA Rules not processed for user `a7c77c879fda42a5907b53c36dfb9dab`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:37.779498 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-683989b1-8964-4474-8b4e-c31ced0c2020 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:37.856692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1300/2600] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:37 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:37.862024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d8fc43-7f59-4ad2-8390-49ec599a71b3 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.899886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:37.930696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683989b1-8964-4474-8b4e-c31ced0c2020 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:37.933178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-683989b1-8964-4474-8b4e-c31ced0c2020 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:37.936655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d8fc43-7f59-4ad2-8390-49ec599a71b3 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.942169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9f5157-428a-4755-b33a-6dadcc63cdc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.943040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9f5157-428a-4755-b33a-6dadcc63cdc6 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.943040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9f5157-428a-4755-b33a-6dadcc63cdc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.943299 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9f5157-428a-4755-b33a-6dadcc63cdc6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.943915 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9f5157-428a-4755-b33a-6dadcc63cdc6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.945859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9f5157-428a-4755-b33a-6dadcc63cdc6 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.957088 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683989b1-8964-4474-8b4e-c31ced0c2020 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:37.962341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:37.962989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:37.963267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:37.963591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] PATH_INFO: `/v3/projects/ff00969c7f524d60beb88f1d85f18073` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:37.964903 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff00969c7f524d60beb88f1d85f18073)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:37.967078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:37.988364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1301/2601] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:37 2022] POST /identity/v3/projects => generated 367 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:40:37.989697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9f5157-428a-4755-b33a-6dadcc63cdc6 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.016364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff00969c7f524d60beb88f1d85f18073', 'request_id': 'req-eedcd737-e26e-443c-ae13-da0885b1ca23'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:38.022192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff00969c7f524d60beb88f1d85f18073', 'request_id': 'req-eedcd737-e26e-443c-ae13-da0885b1ca23'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:38.030527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff00969c7f524d60beb88f1d85f18073', 'request_id': 'req-eedcd737-e26e-443c-ae13-da0885b1ca23'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:38.042694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] The token cache is being invalidate because project ff00969c7f524d60beb88f1d85f18073 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:38.043341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:38.043879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-eedcd737-e26e-443c-ae13-da0885b1ca23 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:38.045499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1301/2602] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:37 2022] DELETE /identity/v3/projects/ff00969c7f524d60beb88f1d85f18073 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:38.050494 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6382e290-d52f-4f43-8c94-19645235d5d1 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.050744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6382e290-d52f-4f43-8c94-19645235d5d1 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.051013 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6382e290-d52f-4f43-8c94-19645235d5d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.056260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9f5157-428a-4755-b33a-6dadcc63cdc6 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.070908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-6382e290-d52f-4f43-8c94-19645235d5d1 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:38.072104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6382e290-d52f-4f43-8c94-19645235d5d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:38.112526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6056d33e-06ce-4794-ae38-7e196bbdd95a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.113130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6056d33e-06ce-4794-ae38-7e196bbdd95a admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.113405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6056d33e-06ce-4794-ae38-7e196bbdd95a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.113676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6056d33e-06ce-4794-ae38-7e196bbdd95a admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.114323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6056d33e-06ce-4794-ae38-7e196bbdd95a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.116279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6056d33e-06ce-4794-ae38-7e196bbdd95a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.123319 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:38.132593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1302/2603] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:38.141909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b52bc-425f-46da-8361-1b5372cb5e20 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.142220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b52bc-425f-46da-8361-1b5372cb5e20 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.142493 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0b52bc-425f-46da-8361-1b5372cb5e20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.146923 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1302/2604] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:37 2022] POST /identity/v3/users => generated 290 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:40:38.150680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6056d33e-06ce-4794-ae38-7e196bbdd95a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.152645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-6056d33e-06ce-4794-ae38-7e196bbdd95a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:38.163436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-8f0b52bc-425f-46da-8361-1b5372cb5e20 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:38.164633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8f0b52bc-425f-46da-8361-1b5372cb5e20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:38.164786 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6056d33e-06ce-4794-ae38-7e196bbdd95a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.169676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a996fb-7959-462e-ae5e-c185269b2461 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.170349 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a996fb-7959-462e-ae5e-c185269b2461 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.170617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a996fb-7959-462e-ae5e-c185269b2461 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.170864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a996fb-7959-462e-ae5e-c185269b2461 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.171527 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a996fb-7959-462e-ae5e-c185269b2461 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.173191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a996fb-7959-462e-ae5e-c185269b2461 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.174396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1303/2605] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:38.180116 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0b52bc-425f-46da-8361-1b5372cb5e20 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.187515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1303/2606] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:38.191888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a996fb-7959-462e-ae5e-c185269b2461 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.192665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0b52bc-425f-46da-8361-1b5372cb5e20 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.199424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86020c5e-8664-46a2-803d-529a58b6f063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.200577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86020c5e-8664-46a2-803d-529a58b6f063 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.200853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86020c5e-8664-46a2-803d-529a58b6f063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.201120 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86020c5e-8664-46a2-803d-529a58b6f063 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.201796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86020c5e-8664-46a2-803d-529a58b6f063 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:40:38.202096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86020c5e-8664-46a2-803d-529a58b6f063 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.203703 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a996fb-7959-462e-ae5e-c185269b2461 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.204166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86020c5e-8664-46a2-803d-529a58b6f063 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.208454 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4ca3e2-dc73-4099-8cbc-4e0be825d35c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.209102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4ca3e2-dc73-4099-8cbc-4e0be825d35c admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.209401 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4ca3e2-dc73-4099-8cbc-4e0be825d35c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.209674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4ca3e2-dc73-4099-8cbc-4e0be825d35c admin admin] PATH_INFO: `/v3/projects/64733c3a6c5c4d19ba8b182829557817/users/5e9113b78a824d49876aa38569cb3872/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.216135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1304/2607] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:40:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:40:38.221147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86020c5e-8664-46a2-803d-529a58b6f063 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.226256 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4ca3e2-dc73-4099-8cbc-4e0be825d35c admin admin] RBAC: Authorizing `identity:create_grant(project_id=64733c3a6c5c4d19ba8b182829557817, user_id=5e9113b78a824d49876aa38569cb3872, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.228908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4ca3e2-dc73-4099-8cbc-4e0be825d35c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.231563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86020c5e-8664-46a2-803d-529a58b6f063 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.236488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceced1f4-b144-417c-8b89-55e6fbf91eb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.237085 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceced1f4-b144-417c-8b89-55e6fbf91eb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.237362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceced1f4-b144-417c-8b89-55e6fbf91eb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.237630 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceced1f4-b144-417c-8b89-55e6fbf91eb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.238304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceced1f4-b144-417c-8b89-55e6fbf91eb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.240258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceced1f4-b144-417c-8b89-55e6fbf91eb0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.242999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1304/2608] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:38 2022] PUT /identity/v3/projects/64733c3a6c5c4d19ba8b182829557817/users/5e9113b78a824d49876aa38569cb3872/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:38.246973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4ca3e2-dc73-4099-8cbc-4e0be825d35c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.267831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:38.272988 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1305/2609] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:38 2022] POST /identity/v3/projects => generated 399 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:38.273978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceced1f4-b144-417c-8b89-55e6fbf91eb0 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.309720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4ca3e2-dc73-4099-8cbc-4e0be825d35c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.337055 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceced1f4-b144-417c-8b89-55e6fbf91eb0 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.341656 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a1d9ab-96f4-4164-83ad-c86f92b49e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.342293 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a1d9ab-96f4-4164-83ad-c86f92b49e77 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.342567 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a1d9ab-96f4-4164-83ad-c86f92b49e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.342836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a1d9ab-96f4-4164-83ad-c86f92b49e77 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.343464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a1d9ab-96f4-4164-83ad-c86f92b49e77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.345377 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a1d9ab-96f4-4164-83ad-c86f92b49e77 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.369635 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1306/2610] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:38 2022] POST /identity/v3/users => generated 300 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:38.369882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0f02a5-f52d-4920-98d9-06c8e288a001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.369882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0f02a5-f52d-4920-98d9-06c8e288a001 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.369882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0f02a5-f52d-4920-98d9-06c8e288a001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.369882 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0f02a5-f52d-4920-98d9-06c8e288a001 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.370412 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0f02a5-f52d-4920-98d9-06c8e288a001 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.370814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a1d9ab-96f4-4164-83ad-c86f92b49e77 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.371971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0f02a5-f52d-4920-98d9-06c8e288a001 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.380247 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a1d9ab-96f4-4164-83ad-c86f92b49e77 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.382566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1305/2611] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:38.384882 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc36a88-1c8d-4259-9f5a-1b8e658d483a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.385564 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc36a88-1c8d-4259-9f5a-1b8e658d483a admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.385961 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc36a88-1c8d-4259-9f5a-1b8e658d483a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.386262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc36a88-1c8d-4259-9f5a-1b8e658d483a admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.386647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0f02a5-f52d-4920-98d9-06c8e288a001 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.386863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc36a88-1c8d-4259-9f5a-1b8e658d483a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.388495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc36a88-1c8d-4259-9f5a-1b8e658d483a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.396650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0f02a5-f52d-4920-98d9-06c8e288a001 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.399040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1307/2612] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:38.403523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc36a88-1c8d-4259-9f5a-1b8e658d483a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.405022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d889afb-8db8-4b3b-8cb1-5ee561331b85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.405685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d889afb-8db8-4b3b-8cb1-5ee561331b85 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.405940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d889afb-8db8-4b3b-8cb1-5ee561331b85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.406251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d889afb-8db8-4b3b-8cb1-5ee561331b85 admin admin] PATH_INFO: `/v3/projects/64733c3a6c5c4d19ba8b182829557817/users/5e9113b78a824d49876aa38569cb3872/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.408395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d889afb-8db8-4b3b-8cb1-5ee561331b85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64733c3a6c5c4d19ba8b182829557817, user_id=5e9113b78a824d49876aa38569cb3872, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.410925 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d889afb-8db8-4b3b-8cb1-5ee561331b85 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.413095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc36a88-1c8d-4259-9f5a-1b8e658d483a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.417699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef129177-a362-4f8d-87b5-74c36bcbf910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.418308 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef129177-a362-4f8d-87b5-74c36bcbf910 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.418582 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef129177-a362-4f8d-87b5-74c36bcbf910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.418849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef129177-a362-4f8d-87b5-74c36bcbf910 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.420096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef129177-a362-4f8d-87b5-74c36bcbf910 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.421655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef129177-a362-4f8d-87b5-74c36bcbf910 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.421971 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1306/2613] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:38 2022] PUT /identity/v3/projects/64733c3a6c5c4d19ba8b182829557817/users/5e9113b78a824d49876aa38569cb3872/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:38.432400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1308/2614] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:38.432611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b1f9a-bbf9-4117-8f50-044983ed8d0f None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.432910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b1f9a-bbf9-4117-8f50-044983ed8d0f None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.433427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60b1f9a-bbf9-4117-8f50-044983ed8d0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.436379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef129177-a362-4f8d-87b5-74c36bcbf910 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.446663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef129177-a362-4f8d-87b5-74c36bcbf910 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.461217 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-f60b1f9a-bbf9-4117-8f50-044983ed8d0f None None] MFA Rules not processed for user `5e9113b78a824d49876aa38569cb3872`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:38.462512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f60b1f9a-bbf9-4117-8f50-044983ed8d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:38.503119 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef3a494-ab94-4a9d-9dd8-92f1bda8d58d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.503781 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef3a494-ab94-4a9d-9dd8-92f1bda8d58d admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.504062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef3a494-ab94-4a9d-9dd8-92f1bda8d58d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.504330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef3a494-ab94-4a9d-9dd8-92f1bda8d58d admin admin] PATH_INFO: `/v3/projects/52fa3c74ab064e73be4a9684c810a91e/users/fc391623c17f4df7b7e12cd55d1aa76f/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.519855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef3a494-ab94-4a9d-9dd8-92f1bda8d58d admin admin] RBAC: Authorizing `identity:create_grant(project_id=52fa3c74ab064e73be4a9684c810a91e, user_id=fc391623c17f4df7b7e12cd55d1aa76f, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.522586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef3a494-ab94-4a9d-9dd8-92f1bda8d58d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.534269 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1307/2615] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:38 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:38.536528 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1309/2616] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:38 2022] PUT /identity/v3/projects/52fa3c74ab064e73be4a9684c810a91e/users/fc391623c17f4df7b7e12cd55d1aa76f/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:38.546834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd5e01-aa1e-4d15-8fb8-cc2fd2eb8f3e None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.547007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef3a494-ab94-4a9d-9dd8-92f1bda8d58d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.547116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd5e01-aa1e-4d15-8fb8-cc2fd2eb8f3e None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.547380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd5e01-aa1e-4d15-8fb8-cc2fd2eb8f3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.548659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6ef3a494-ab94-4a9d-9dd8-92f1bda8d58d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:38.560213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:38.575822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-88bd5e01-aa1e-4d15-8fb8-cc2fd2eb8f3e None None] MFA Rules not processed for user `fc391623c17f4df7b7e12cd55d1aa76f`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:38.577035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-88bd5e01-aa1e-4d15-8fb8-cc2fd2eb8f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:38.600671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:38.608620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef3a494-ab94-4a9d-9dd8-92f1bda8d58d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.612480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc2569f-df5c-4a8d-8db3-de77b81fe304 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.613086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc2569f-df5c-4a8d-8db3-de77b81fe304 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.613362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc2569f-df5c-4a8d-8db3-de77b81fe304 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.613705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc2569f-df5c-4a8d-8db3-de77b81fe304 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.649459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1308/2617] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:38 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:38.654284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bd5e01-aa1e-4d15-8fb8-cc2fd2eb8f3e None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.674939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc2569f-df5c-4a8d-8db3-de77b81fe304 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.676454 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc2569f-df5c-4a8d-8db3-de77b81fe304 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.686368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1310/2618] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:38 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:40:38.720529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bd5e01-aa1e-4d15-8fb8-cc2fd2eb8f3e None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.725363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f6b85b-7106-44a0-9d7c-5a8f1eb2ca5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.725980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f6b85b-7106-44a0-9d7c-5a8f1eb2ca5d admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.726327 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f6b85b-7106-44a0-9d7c-5a8f1eb2ca5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.726597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f6b85b-7106-44a0-9d7c-5a8f1eb2ca5d admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.727231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f6b85b-7106-44a0-9d7c-5a8f1eb2ca5d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.729335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f6b85b-7106-44a0-9d7c-5a8f1eb2ca5d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.734359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc2569f-df5c-4a8d-8db3-de77b81fe304 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.735922 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-bcc2569f-df5c-4a8d-8db3-de77b81fe304 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:38.747266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc2569f-df5c-4a8d-8db3-de77b81fe304 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.751109 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625067b8-8d86-4298-8068-5292ca9a3816 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.751709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625067b8-8d86-4298-8068-5292ca9a3816 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.751980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625067b8-8d86-4298-8068-5292ca9a3816 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.752242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625067b8-8d86-4298-8068-5292ca9a3816 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.754314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-625067b8-8d86-4298-8068-5292ca9a3816 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:38.759601 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1309/2619] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:38 2022] POST /identity/v3/projects => generated 399 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:40:38.763962 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f6b85b-7106-44a0-9d7c-5a8f1eb2ca5d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.812546 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625067b8-8d86-4298-8068-5292ca9a3816 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.813985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625067b8-8d86-4298-8068-5292ca9a3816 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.822692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1311/2620] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:38 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:38.825048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f6b85b-7106-44a0-9d7c-5a8f1eb2ca5d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.830167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e259e3-f4b0-44ea-9aa4-5ed077192643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.830775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e259e3-f4b0-44ea-9aa4-5ed077192643 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.831043 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e259e3-f4b0-44ea-9aa4-5ed077192643 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.831312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e259e3-f4b0-44ea-9aa4-5ed077192643 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.831940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e259e3-f4b0-44ea-9aa4-5ed077192643 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.833852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e259e3-f4b0-44ea-9aa4-5ed077192643 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.856518 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1310/2621] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:38 2022] POST /identity/v3/users => generated 306 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:40:38.860867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625067b8-8d86-4298-8068-5292ca9a3816 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.871689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625067b8-8d86-4298-8068-5292ca9a3816 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.876758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ac250f-03a7-43dc-9c60-d54bccf02226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.877370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ac250f-03a7-43dc-9c60-d54bccf02226 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.877647 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ac250f-03a7-43dc-9c60-d54bccf02226 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.877922 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ac250f-03a7-43dc-9c60-d54bccf02226 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.878590 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ac250f-03a7-43dc-9c60-d54bccf02226 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.880178 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ac250f-03a7-43dc-9c60-d54bccf02226 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.890276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1312/2622] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:38.894548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e259e3-f4b0-44ea-9aa4-5ed077192643 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.904592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e259e3-f4b0-44ea-9aa4-5ed077192643 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:38.909123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c235bf-e7f4-4ec1-b7d0-f9d273935893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:38.909779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c235bf-e7f4-4ec1-b7d0-f9d273935893 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:38.910061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c235bf-e7f4-4ec1-b7d0-f9d273935893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:38.910427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c235bf-e7f4-4ec1-b7d0-f9d273935893 admin admin] PATH_INFO: `/v3/projects/2a42416eeb3d4229b7cb6719ee9cfa9c/users/2c4116a2ba5b4db28fc430e27f2245f8/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:38.928490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c235bf-e7f4-4ec1-b7d0-f9d273935893 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a42416eeb3d4229b7cb6719ee9cfa9c, user_id=2c4116a2ba5b4db28fc430e27f2245f8, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:38.931354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c235bf-e7f4-4ec1-b7d0-f9d273935893 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:38.946142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1311/2623] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:38 2022] PUT /identity/v3/projects/2a42416eeb3d4229b7cb6719ee9cfa9c/users/2c4116a2ba5b4db28fc430e27f2245f8/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:38.950881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ac250f-03a7-43dc-9c60-d54bccf02226 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:38.962043 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ac250f-03a7-43dc-9c60-d54bccf02226 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:39.012227 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:39.026395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554f5a0a-0807-4efc-8095-4ff00a114b6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:39.027079 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554f5a0a-0807-4efc-8095-4ff00a114b6c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:39.027359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554f5a0a-0807-4efc-8095-4ff00a114b6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:39.027814 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554f5a0a-0807-4efc-8095-4ff00a114b6c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:39.028578 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554f5a0a-0807-4efc-8095-4ff00a114b6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:39.030324 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554f5a0a-0807-4efc-8095-4ff00a114b6c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:39.041489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1313/2624] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:38 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:39.045953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c235bf-e7f4-4ec1-b7d0-f9d273935893 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:39.056648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c235bf-e7f4-4ec1-b7d0-f9d273935893 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:39.062873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de60986-4481-4ecf-bef1-4c88b8825de8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:39.063576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de60986-4481-4ecf-bef1-4c88b8825de8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:39.063912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de60986-4481-4ecf-bef1-4c88b8825de8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:39.064353 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de60986-4481-4ecf-bef1-4c88b8825de8 admin admin] PATH_INFO: `/v3/projects/2a42416eeb3d4229b7cb6719ee9cfa9c/users/2c4116a2ba5b4db28fc430e27f2245f8/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:39.066762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de60986-4481-4ecf-bef1-4c88b8825de8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a42416eeb3d4229b7cb6719ee9cfa9c, user_id=2c4116a2ba5b4db28fc430e27f2245f8, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:39.069658 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de60986-4481-4ecf-bef1-4c88b8825de8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:39.082634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1312/2625] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:39 2022] PUT /identity/v3/projects/2a42416eeb3d4229b7cb6719ee9cfa9c/users/2c4116a2ba5b4db28fc430e27f2245f8/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:40:39.093152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49159386-c0bd-4121-bf6c-d512181efa41 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:39.093442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49159386-c0bd-4121-bf6c-d512181efa41 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:39.093721 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49159386-c0bd-4121-bf6c-d512181efa41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:39.125968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-49159386-c0bd-4121-bf6c-d512181efa41 None None] MFA Rules not processed for user `2c4116a2ba5b4db28fc430e27f2245f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:39.138741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-49159386-c0bd-4121-bf6c-d512181efa41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:39.198752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1314/2626] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:39 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:39.210825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de60986-4481-4ecf-bef1-4c88b8825de8 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:39.216297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:39.278863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de60986-4481-4ecf-bef1-4c88b8825de8 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:39.283546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa789580-68e7-4f92-86d3-042633557602 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:39.284161 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa789580-68e7-4f92-86d3-042633557602 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:39.284436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa789580-68e7-4f92-86d3-042633557602 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:39.284702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa789580-68e7-4f92-86d3-042633557602 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:39.294769 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa789580-68e7-4f92-86d3-042633557602 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:39.296225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa789580-68e7-4f92-86d3-042633557602 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:39.305873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1313/2627] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:39 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:39.390997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49159386-c0bd-4121-bf6c-d512181efa41 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:39.401245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49159386-c0bd-4121-bf6c-d512181efa41 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:39.405276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86310196-f0fe-432f-9242-1b0e256547ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:39.405871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86310196-f0fe-432f-9242-1b0e256547ba service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:39.406180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86310196-f0fe-432f-9242-1b0e256547ba service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:39.406444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86310196-f0fe-432f-9242-1b0e256547ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:39.463202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86310196-f0fe-432f-9242-1b0e256547ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:39.464561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86310196-f0fe-432f-9242-1b0e256547ba service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:39.473485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1315/2628] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:39 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:42.258907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa789580-68e7-4f92-86d3-042633557602 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:42.326979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa789580-68e7-4f92-86d3-042633557602 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:42.331539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:42.332179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:42.332453 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:42.332723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] PATH_INFO: `/v3/users/a7c77c879fda42a5907b53c36dfb9dab` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:42.333842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7c77c879fda42a5907b53c36dfb9dab)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:42.335811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:42.392976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7c77c879fda42a5907b53c36dfb9dab'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:42.393707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7c77c879fda42a5907b53c36dfb9dab'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:42.402994 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7c77c879fda42a5907b53c36dfb9dab'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:42.403710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7c77c879fda42a5907b53c36dfb9dab'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:42.413434 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7c77c879fda42a5907b53c36dfb9dab'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:42.432836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1314/2629] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:42 2022] DELETE /identity/v3/users/a7c77c879fda42a5907b53c36dfb9dab => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:42.438339 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d4cb52-e61e-4734-a72f-a1b9db0b1ac4 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:42.438906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d4cb52-e61e-4734-a72f-a1b9db0b1ac4 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:42.438906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d4cb52-e61e-4734-a72f-a1b9db0b1ac4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:42.462871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e5d4cb52-e61e-4734-a72f-a1b9db0b1ac4 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:42.464080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e5d4cb52-e61e-4734-a72f-a1b9db0b1ac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:42.529242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1316/2630] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:42.539556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:42.541318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:42.572010 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:42.605278 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34d7398-50b1-4da5-bb3d-b5f8b95f282a admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:42.609817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a1c151-3b1b-49db-982c-ddc76ccb9688 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:42.610547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a1c151-3b1b-49db-982c-ddc76ccb9688 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:42.610751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a1c151-3b1b-49db-982c-ddc76ccb9688 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:42.611026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a1c151-3b1b-49db-982c-ddc76ccb9688 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:42.620323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a1c151-3b1b-49db-982c-ddc76ccb9688 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:42.621940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a1c151-3b1b-49db-982c-ddc76ccb9688 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:42.631664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1315/2631] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:42.909960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d4cb52-e61e-4734-a72f-a1b9db0b1ac4 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:42.911545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e5d4cb52-e61e-4734-a72f-a1b9db0b1ac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:42.923943 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d4cb52-e61e-4734-a72f-a1b9db0b1ac4 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:42.928655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:42.929290 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:42.929563 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:42.929837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] PATH_INFO: `/v3/projects/29252a7765dc4332880995290f130f9d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:42.930998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] RBAC: Authorizing `identity:delete_project(project_id=29252a7765dc4332880995290f130f9d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:42.933584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:42.935343 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:42.988364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29252a7765dc4332880995290f130f9d', 'request_id': 'req-f883188f-d59a-415d-b735-78373d4c2d05'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:42.994266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29252a7765dc4332880995290f130f9d', 'request_id': 'req-f883188f-d59a-415d-b735-78373d4c2d05'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.004335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29252a7765dc4332880995290f130f9d', 'request_id': 'req-f883188f-d59a-415d-b735-78373d4c2d05'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.016111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] The token cache is being invalidate because project 29252a7765dc4332880995290f130f9d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:43.016391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.017007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.018522 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1317/2632] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:42 2022] DELETE /identity/v3/projects/29252a7765dc4332880995290f130f9d => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:43.022721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a1c151-3b1b-49db-982c-ddc76ccb9688 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:43.032627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a1c151-3b1b-49db-982c-ddc76ccb9688 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:43.089588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:43.090273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:43.090550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:43.090818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] PATH_INFO: `/v3/users/5e9113b78a824d49876aa38569cb3872` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:43.092433 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e9113b78a824d49876aa38569cb3872)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:43.094068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:43.142274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e9113b78a824d49876aa38569cb3872'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.142924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e9113b78a824d49876aa38569cb3872'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.150048 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e9113b78a824d49876aa38569cb3872'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.150714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e9113b78a824d49876aa38569cb3872'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.160691 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-38b1b531-bcf0-4e69-a241-d354b82ba50e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e9113b78a824d49876aa38569cb3872'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.187045 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1316/2633] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:43 2022] DELETE /identity/v3/users/5e9113b78a824d49876aa38569cb3872 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:43.473866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:43.475899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:43.494829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f883188f-d59a-415d-b735-78373d4c2d05 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:43.557347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:43.558000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:43.558312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:43.558589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] PATH_INFO: `/v3/projects/64733c3a6c5c4d19ba8b182829557817` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:43.559718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=64733c3a6c5c4d19ba8b182829557817)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:43.561663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:43.613618 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64733c3a6c5c4d19ba8b182829557817', 'request_id': 'req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.619779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64733c3a6c5c4d19ba8b182829557817', 'request_id': 'req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.630147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64733c3a6c5c4d19ba8b182829557817', 'request_id': 'req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.640768 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] The token cache is being invalidate because project 64733c3a6c5c4d19ba8b182829557817 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:43.641060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.641782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.643652 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1318/2634] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:43 2022] DELETE /identity/v3/projects/64733c3a6c5c4d19ba8b182829557817 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:43.648207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8c7c07-43f9-4648-9d3f-50ec49f558b5 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:43.648660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8c7c07-43f9-4648-9d3f-50ec49f558b5 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:43.649046 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8c7c07-43f9-4648-9d3f-50ec49f558b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:43.650318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:43.651410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:43.672316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7f8c7c07-43f9-4648-9d3f-50ec49f558b5 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:43.672894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:43.673520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7f8c7c07-43f9-4648-9d3f-50ec49f558b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:43.717486 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:43.720934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b97c7-55f6-4ede-9234-f3c2f27c77a1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:43.725908 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:43.726581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:43.726863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:43.727139 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] PATH_INFO: `/v3/users/fc391623c17f4df7b7e12cd55d1aa76f` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:43.728284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc391623c17f4df7b7e12cd55d1aa76f)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:43.730370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:43.736771 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1317/2635] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:43.744700 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3035f2ab-634d-4285-8158-a6c9dcea7a64 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:43.744989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3035f2ab-634d-4285-8158-a6c9dcea7a64 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:43.745263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3035f2ab-634d-4285-8158-a6c9dcea7a64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:43.765462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3035f2ab-634d-4285-8158-a6c9dcea7a64 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:43.766719 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3035f2ab-634d-4285-8158-a6c9dcea7a64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:43.778866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1318/2636] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:43.780038 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3035f2ab-634d-4285-8158-a6c9dcea7a64 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:43.780849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc391623c17f4df7b7e12cd55d1aa76f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.789021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3035f2ab-634d-4285-8158-a6c9dcea7a64 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:43.790015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc391623c17f4df7b7e12cd55d1aa76f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.790845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc391623c17f4df7b7e12cd55d1aa76f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.791592 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc391623c17f4df7b7e12cd55d1aa76f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.793724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c568d8b-0f1c-4813-90ec-1ad19f715085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:43.794350 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c568d8b-0f1c-4813-90ec-1ad19f715085 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:43.794670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c568d8b-0f1c-4813-90ec-1ad19f715085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:43.794946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c568d8b-0f1c-4813-90ec-1ad19f715085 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:43.795593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c568d8b-0f1c-4813-90ec-1ad19f715085 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:40:43.795895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c568d8b-0f1c-4813-90ec-1ad19f715085 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:43.797502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c568d8b-0f1c-4813-90ec-1ad19f715085 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:43.799695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-05217673-3694-499c-9bf4-7164c4a9dcf6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc391623c17f4df7b7e12cd55d1aa76f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:43.809919 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1319/2637] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:40:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:40:43.814723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c568d8b-0f1c-4813-90ec-1ad19f715085 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:43.816364 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4c568d8b-0f1c-4813-90ec-1ad19f715085 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:43.818345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1319/2638] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:43 2022] DELETE /identity/v3/users/fc391623c17f4df7b7e12cd55d1aa76f => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:40:43.822594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59b1a44-834d-4120-9582-be2f58e6be36 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:43.822950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59b1a44-834d-4120-9582-be2f58e6be36 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:43.823332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59b1a44-834d-4120-9582-be2f58e6be36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:43.829496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c568d8b-0f1c-4813-90ec-1ad19f715085 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:43.842876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-c59b1a44-834d-4120-9582-be2f58e6be36 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:43.844097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c59b1a44-834d-4120-9582-be2f58e6be36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:43.887868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05719d25-4f83-46e2-be47-42f07fe342a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:43.888881 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05719d25-4f83-46e2-be47-42f07fe342a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:43.889189 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05719d25-4f83-46e2-be47-42f07fe342a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:43.889494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05719d25-4f83-46e2-be47-42f07fe342a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:43.890734 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05719d25-4f83-46e2-be47-42f07fe342a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:43.892743 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05719d25-4f83-46e2-be47-42f07fe342a4 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:43.903920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1320/2639] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:43.915076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59b1a44-834d-4120-9582-be2f58e6be36 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:43.916636 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c59b1a44-834d-4120-9582-be2f58e6be36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:43.927647 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1320/2640] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:43 2022] POST /identity/v3/projects => generated 385 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:43.928636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05719d25-4f83-46e2-be47-42f07fe342a4 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:43.961410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:43.979149 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59b1a44-834d-4120-9582-be2f58e6be36 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.004485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05719d25-4f83-46e2-be47-42f07fe342a4 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.013224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed76de-924a-49b6-8dd8-585353dd876b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.013851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed76de-924a-49b6-8dd8-585353dd876b admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.014238 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed76de-924a-49b6-8dd8-585353dd876b admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.014580 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daed76de-924a-49b6-8dd8-585353dd876b admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.015241 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daed76de-924a-49b6-8dd8-585353dd876b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.017174 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daed76de-924a-49b6-8dd8-585353dd876b admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.030667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba776e50-f0a3-4210-ab9c-f0cafead7b62 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.031367 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba776e50-f0a3-4210-ab9c-f0cafead7b62 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.031730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba776e50-f0a3-4210-ab9c-f0cafead7b62 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.032040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba776e50-f0a3-4210-ab9c-f0cafead7b62 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.042224 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1321/2641] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:43 2022] POST /identity/v3/users => generated 293 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:40:44.042402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba776e50-f0a3-4210-ab9c-f0cafead7b62 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.043874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba776e50-f0a3-4210-ab9c-f0cafead7b62 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.046616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed76de-924a-49b6-8dd8-585353dd876b admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.054680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1321/2642] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:40:44.056694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daed76de-924a-49b6-8dd8-585353dd876b admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.061913 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe5b951-0796-4bc5-a684-12f5ce1197e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.062569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe5b951-0796-4bc5-a684-12f5ce1197e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.062856 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe5b951-0796-4bc5-a684-12f5ce1197e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.063128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe5b951-0796-4bc5-a684-12f5ce1197e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.063775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe5b951-0796-4bc5-a684-12f5ce1197e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.065477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe5b951-0796-4bc5-a684-12f5ce1197e9 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.078155 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1322/2643] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:44.081353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba776e50-f0a3-4210-ab9c-f0cafead7b62 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.091158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba776e50-f0a3-4210-ab9c-f0cafead7b62 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.097160 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf761319-67c7-4f64-9608-f2970346775f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.097873 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf761319-67c7-4f64-9608-f2970346775f admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.098179 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf761319-67c7-4f64-9608-f2970346775f admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.098447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf761319-67c7-4f64-9608-f2970346775f admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.099089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf761319-67c7-4f64-9608-f2970346775f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.100707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf761319-67c7-4f64-9608-f2970346775f admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.111617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1322/2644] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:44.116244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe5b951-0796-4bc5-a684-12f5ce1197e9 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.126684 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe5b951-0796-4bc5-a684-12f5ce1197e9 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.131634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802d28a1-a8a7-41b2-a7e3-508545731129 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.132296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d28a1-a8a7-41b2-a7e3-508545731129 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.132577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d28a1-a8a7-41b2-a7e3-508545731129 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.132852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d28a1-a8a7-41b2-a7e3-508545731129 admin admin] PATH_INFO: `/v3/projects/92c56a4dfb4f41ce8a43082cb9451e30/users/bc114646de92480487f399e4218087bf/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.149374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802d28a1-a8a7-41b2-a7e3-508545731129 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92c56a4dfb4f41ce8a43082cb9451e30, user_id=bc114646de92480487f399e4218087bf, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.152104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802d28a1-a8a7-41b2-a7e3-508545731129 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.167112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1323/2645] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:44 2022] PUT /identity/v3/projects/92c56a4dfb4f41ce8a43082cb9451e30/users/bc114646de92480487f399e4218087bf/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:44.178797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3c1787-9842-4275-a81d-9b2e5f2053a8 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.179517 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3c1787-9842-4275-a81d-9b2e5f2053a8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.179797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3c1787-9842-4275-a81d-9b2e5f2053a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.210982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4e3c1787-9842-4275-a81d-9b2e5f2053a8 None None] MFA Rules not processed for user `bc114646de92480487f399e4218087bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:44.212845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4e3c1787-9842-4275-a81d-9b2e5f2053a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:44.281334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1323/2646] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:44 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:44.286003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802d28a1-a8a7-41b2-a7e3-508545731129 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.295037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:44.353313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3c1787-9842-4275-a81d-9b2e5f2053a8 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.355009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4e3c1787-9842-4275-a81d-9b2e5f2053a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:44.360311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802d28a1-a8a7-41b2-a7e3-508545731129 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.366044 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cb323b-d306-46cf-ae66-094b12f7add2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.366802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cb323b-d306-46cf-ae66-094b12f7add2 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.367312 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cb323b-d306-46cf-ae66-094b12f7add2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.367552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cb323b-d306-46cf-ae66-094b12f7add2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.368314 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cb323b-d306-46cf-ae66-094b12f7add2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.370344 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cb323b-d306-46cf-ae66-094b12f7add2 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.370486 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3c1787-9842-4275-a81d-9b2e5f2053a8 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.381911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.382600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.382874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.383141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] PATH_INFO: `/v3/projects/52fa3c74ab064e73be4a9684c810a91e` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.384284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] RBAC: Authorizing `identity:delete_project(project_id=52fa3c74ab064e73be4a9684c810a91e)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.386219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.405826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1324/2647] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:44 2022] POST /identity/v3/projects => generated 387 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:44.409826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cb323b-d306-46cf-ae66-094b12f7add2 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.439797 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52fa3c74ab064e73be4a9684c810a91e', 'request_id': 'req-b05ef7dd-b546-4829-b48d-c278e37a298a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:44.445370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52fa3c74ab064e73be4a9684c810a91e', 'request_id': 'req-b05ef7dd-b546-4829-b48d-c278e37a298a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:44.455206 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52fa3c74ab064e73be4a9684c810a91e', 'request_id': 'req-b05ef7dd-b546-4829-b48d-c278e37a298a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:44.466485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:44.471632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] The token cache is being invalidate because project 52fa3c74ab064e73be4a9684c810a91e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:44.471931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:44.473198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:44.474863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1324/2648] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:44 2022] DELETE /identity/v3/projects/52fa3c74ab064e73be4a9684c810a91e => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:44.478830 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.479991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cb323b-d306-46cf-ae66-094b12f7add2 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.480701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:44.537524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e878f77b-1ca1-46c0-ac19-6031cc43f1b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.538187 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e878f77b-1ca1-46c0-ac19-6031cc43f1b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.538479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e878f77b-1ca1-46c0-ac19-6031cc43f1b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.538811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e878f77b-1ca1-46c0-ac19-6031cc43f1b1 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.539374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e878f77b-1ca1-46c0-ac19-6031cc43f1b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.541602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e878f77b-1ca1-46c0-ac19-6031cc43f1b1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.547990 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05ef7dd-b546-4829-b48d-c278e37a298a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.553115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.553861 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.554093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.554393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] PATH_INFO: `/v3/users/2c4116a2ba5b4db28fc430e27f2245f8` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.555576 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c4116a2ba5b4db28fc430e27f2245f8)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.557485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.566138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1325/2649] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:44 2022] POST /identity/v3/users => generated 300 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:40:44.569109 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e878f77b-1ca1-46c0-ac19-6031cc43f1b1 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.570832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e878f77b-1ca1-46c0-ac19-6031cc43f1b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:44.584697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e878f77b-1ca1-46c0-ac19-6031cc43f1b1 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.589745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e597dc3a-62e7-426e-b3db-99cc554c06ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.590379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e597dc3a-62e7-426e-b3db-99cc554c06ae admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.590673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e597dc3a-62e7-426e-b3db-99cc554c06ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.590947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e597dc3a-62e7-426e-b3db-99cc554c06ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.591614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e597dc3a-62e7-426e-b3db-99cc554c06ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.593497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e597dc3a-62e7-426e-b3db-99cc554c06ae admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.604637 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1326/2650] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:44 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:44.610572 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e597dc3a-62e7-426e-b3db-99cc554c06ae admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.615318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c4116a2ba5b4db28fc430e27f2245f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:44.619955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e597dc3a-62e7-426e-b3db-99cc554c06ae admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.623530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c4116a2ba5b4db28fc430e27f2245f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:44.624244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c4116a2ba5b4db28fc430e27f2245f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:44.625038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c4116a2ba5b4db28fc430e27f2245f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:44.625377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016fd30-12ac-429f-8961-be4cb8307bde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.626019 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016fd30-12ac-429f-8961-be4cb8307bde admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.626802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016fd30-12ac-429f-8961-be4cb8307bde admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.627078 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016fd30-12ac-429f-8961-be4cb8307bde admin admin] PATH_INFO: `/v3/projects/cb6b544858c649acbf0c7292840b60e5/users/6d5680b71f0f42e0bf6e871b9bd76e34/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.632670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c4116a2ba5b4db28fc430e27f2245f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:44.646952 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b016fd30-12ac-429f-8961-be4cb8307bde admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb6b544858c649acbf0c7292840b60e5, user_id=6d5680b71f0f42e0bf6e871b9bd76e34, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.649444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b016fd30-12ac-429f-8961-be4cb8307bde admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.651705 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:44.659444 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1325/2651] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:44 2022] DELETE /identity/v3/users/2c4116a2ba5b4db28fc430e27f2245f8 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:44.668644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1327/2652] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:44 2022] PUT /identity/v3/projects/cb6b544858c649acbf0c7292840b60e5/users/6d5680b71f0f42e0bf6e871b9bd76e34/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:44.672916 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.674485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:44.744363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d21e9cb-482b-4f02-b935-088b391a6b7e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.769244 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:44.805921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b05bb9-d0cd-4fe1-9fe5-14606a6861fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.806858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b05bb9-d0cd-4fe1-9fe5-14606a6861fc admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.807137 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b05bb9-d0cd-4fe1-9fe5-14606a6861fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.807409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b05bb9-d0cd-4fe1-9fe5-14606a6861fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.808462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b05bb9-d0cd-4fe1-9fe5-14606a6861fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.809965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b05bb9-d0cd-4fe1-9fe5-14606a6861fc admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.820773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1326/2653] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:44 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:44.824982 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016fd30-12ac-429f-8961-be4cb8307bde admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.836508 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016fd30-12ac-429f-8961-be4cb8307bde admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:44.841211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08c95a8-b5c0-42be-b274-8fb5d1a1d0ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:44.841959 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08c95a8-b5c0-42be-b274-8fb5d1a1d0ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.842335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08c95a8-b5c0-42be-b274-8fb5d1a1d0ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.842611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08c95a8-b5c0-42be-b274-8fb5d1a1d0ab admin admin] PATH_INFO: `/v3/projects/cb6b544858c649acbf0c7292840b60e5/users/6d5680b71f0f42e0bf6e871b9bd76e34/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.844626 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08c95a8-b5c0-42be-b274-8fb5d1a1d0ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb6b544858c649acbf0c7292840b60e5, user_id=6d5680b71f0f42e0bf6e871b9bd76e34, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:44.847309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08c95a8-b5c0-42be-b274-8fb5d1a1d0ab admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:44.861128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1328/2654] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:44 2022] PUT /identity/v3/projects/cb6b544858c649acbf0c7292840b60e5/users/6d5680b71f0f42e0bf6e871b9bd76e34/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:44.869960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0460a8-60c1-4807-98e2-c1bc097fb4c2 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:44.870374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0460a8-60c1-4807-98e2-c1bc097fb4c2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:44.870727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0460a8-60c1-4807-98e2-c1bc097fb4c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:44.900021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-cf0460a8-60c1-4807-98e2-c1bc097fb4c2 None None] MFA Rules not processed for user `6d5680b71f0f42e0bf6e871b9bd76e34`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:44.901241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-cf0460a8-60c1-4807-98e2-c1bc097fb4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:44.932452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08c95a8-b5c0-42be-b274-8fb5d1a1d0ab admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.934370 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a08c95a8-b5c0-42be-b274-8fb5d1a1d0ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:44.983105 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1327/2655] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:44 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:44.995905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0460a8-60c1-4807-98e2-c1bc097fb4c2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:44.997268 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-cf0460a8-60c1-4807-98e2-c1bc097fb4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:45.012983 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08c95a8-b5c0-42be-b274-8fb5d1a1d0ab admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.018331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.018710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.019023 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.019291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] PATH_INFO: `/v3/projects/2a42416eeb3d4229b7cb6719ee9cfa9c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.019291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a42416eeb3d4229b7cb6719ee9cfa9c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.022500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.058129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0460a8-60c1-4807-98e2-c1bc097fb4c2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.062456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae078f3-cec4-45ff-958a-d5d900a02cc1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.063066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae078f3-cec4-45ff-958a-d5d900a02cc1 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.063361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae078f3-cec4-45ff-958a-d5d900a02cc1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.063791 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae078f3-cec4-45ff-958a-d5d900a02cc1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.074480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae078f3-cec4-45ff-958a-d5d900a02cc1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.075940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae078f3-cec4-45ff-958a-d5d900a02cc1 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.078516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a42416eeb3d4229b7cb6719ee9cfa9c', 'request_id': 'req-653e560e-ca17-48b3-bf30-70f3282680f8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:45.084997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a42416eeb3d4229b7cb6719ee9cfa9c', 'request_id': 'req-653e560e-ca17-48b3-bf30-70f3282680f8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:45.089073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1328/2656] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:44 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:40:45.095530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a42416eeb3d4229b7cb6719ee9cfa9c', 'request_id': 'req-653e560e-ca17-48b3-bf30-70f3282680f8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:45.111272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] The token cache is being invalidate because project 2a42416eeb3d4229b7cb6719ee9cfa9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:45.111558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:45.112336 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:45.117784 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1329/2657] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:44 2022] DELETE /identity/v3/projects/2a42416eeb3d4229b7cb6719ee9cfa9c => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:40:45.120027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0410a7f-f393-4fd1-9577-4b0e2120074a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.120311 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0410a7f-f393-4fd1-9577-4b0e2120074a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.120754 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0410a7f-f393-4fd1-9577-4b0e2120074a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.145211 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a0410a7f-f393-4fd1-9577-4b0e2120074a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:45.146449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a0410a7f-f393-4fd1-9577-4b0e2120074a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:45.175659 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.177235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:45.201852 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=100 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:45.217322 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1329/2658] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:45.227158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b1ac60-183e-405b-b6e2-04547fd705a8 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.227445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b1ac60-183e-405b-b6e2-04547fd705a8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.227720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b1ac60-183e-405b-b6e2-04547fd705a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.248539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-65b1ac60-183e-405b-b6e2-04547fd705a8 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:45.249720 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-65b1ac60-183e-405b-b6e2-04547fd705a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:45.257303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653e560e-ca17-48b3-bf30-70f3282680f8 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.257870 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1330/2659] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:45.262173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.262794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.262910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b1ac60-183e-405b-b6e2-04547fd705a8 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.263068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.263335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] PATH_INFO: `/v3/users/bc114646de92480487f399e4218087bf` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.264706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc114646de92480487f399e4218087bf)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.266722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.274759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b1ac60-183e-405b-b6e2-04547fd705a8 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.280458 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:INSERT=3,keystone:SELECT=89,keystone:DELETE=8 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:45.280685 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d219ed6-7eb4-4120-8281-0be08a8511c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.281298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d219ed6-7eb4-4120-8281-0be08a8511c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.281584 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d219ed6-7eb4-4120-8281-0be08a8511c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.281883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d219ed6-7eb4-4120-8281-0be08a8511c1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.282550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d219ed6-7eb4-4120-8281-0be08a8511c1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:40:45.283491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d219ed6-7eb4-4120-8281-0be08a8511c1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.284973 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d219ed6-7eb4-4120-8281-0be08a8511c1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.300440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1331/2660] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:40:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:40:45.304917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d219ed6-7eb4-4120-8281-0be08a8511c1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.317530 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d219ed6-7eb4-4120-8281-0be08a8511c1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.322839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6b3848-853a-44e3-87e3-b7a471147fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.323867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b3848-853a-44e3-87e3-b7a471147fa6 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.324341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b3848-853a-44e3-87e3-b7a471147fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.324802 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6b3848-853a-44e3-87e3-b7a471147fa6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.325989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6b3848-853a-44e3-87e3-b7a471147fa6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.327444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc114646de92480487f399e4218087bf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:45.328381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc114646de92480487f399e4218087bf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:45.329344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6b3848-853a-44e3-87e3-b7a471147fa6 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.335974 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc114646de92480487f399e4218087bf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:45.336805 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc114646de92480487f399e4218087bf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:45.345088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-51897ba0-e084-4154-9a23-799515801327 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc114646de92480487f399e4218087bf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:45.360892 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1332/2661] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:45 2022] POST /identity/v3/projects => generated 377 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:45.365158 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6b3848-853a-44e3-87e3-b7a471147fa6 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.367186 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-db6b3848-853a-44e3-87e3-b7a471147fa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:45.368035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1330/2662] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:45 2022] DELETE /identity/v3/users/bc114646de92480487f399e4218087bf => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:45.375528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3df93c-f635-4afa-bc13-ee786ec74e9a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.375827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3df93c-f635-4afa-bc13-ee786ec74e9a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.376102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3df93c-f635-4afa-bc13-ee786ec74e9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.395976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7b3df93c-f635-4afa-bc13-ee786ec74e9a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:45.397471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7b3df93c-f635-4afa-bc13-ee786ec74e9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:45.445602 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6b3848-853a-44e3-87e3-b7a471147fa6 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.490210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1331/2663] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:45.499264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3df93c-f635-4afa-bc13-ee786ec74e9a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.501842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7b3df93c-f635-4afa-bc13-ee786ec74e9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:45.510362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3caab593-3e12-4697-9a93-5d1cc2457901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.510991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caab593-3e12-4697-9a93-5d1cc2457901 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.511262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caab593-3e12-4697-9a93-5d1cc2457901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.511519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caab593-3e12-4697-9a93-5d1cc2457901 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.512470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3caab593-3e12-4697-9a93-5d1cc2457901 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.515073 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3caab593-3e12-4697-9a93-5d1cc2457901 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.541397 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1333/2664] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:45 2022] POST /identity/v3/users => generated 289 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:45.546210 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3caab593-3e12-4697-9a93-5d1cc2457901 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.556817 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3caab593-3e12-4697-9a93-5d1cc2457901 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.562979 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8a675-305d-4251-a8b9-632d98b74cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.563674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8a675-305d-4251-a8b9-632d98b74cac admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.564015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8a675-305d-4251-a8b9-632d98b74cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.564408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8a675-305d-4251-a8b9-632d98b74cac admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.565096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d8a675-305d-4251-a8b9-632d98b74cac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.566924 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d8a675-305d-4251-a8b9-632d98b74cac admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.578663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1334/2665] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:45 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:45.580654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3df93c-f635-4afa-bc13-ee786ec74e9a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.583696 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8a675-305d-4251-a8b9-632d98b74cac admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.585496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aca89fb-bb81-4def-baa8-331d669e6694 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.586149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aca89fb-bb81-4def-baa8-331d669e6694 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.586432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aca89fb-bb81-4def-baa8-331d669e6694 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.586702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aca89fb-bb81-4def-baa8-331d669e6694 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.595932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8a675-305d-4251-a8b9-632d98b74cac admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.597846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aca89fb-bb81-4def-baa8-331d669e6694 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.599464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aca89fb-bb81-4def-baa8-331d669e6694 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.612461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1332/2666] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:45.615429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cf1503-09a0-4781-882f-5e3dbfa74bbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.616075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cf1503-09a0-4781-882f-5e3dbfa74bbc admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.616352 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cf1503-09a0-4781-882f-5e3dbfa74bbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.616626 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cf1503-09a0-4781-882f-5e3dbfa74bbc admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.617272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cf1503-09a0-4781-882f-5e3dbfa74bbc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.618942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cf1503-09a0-4781-882f-5e3dbfa74bbc admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.634235 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1335/2667] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:45 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:45.636007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aca89fb-bb81-4def-baa8-331d669e6694 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.647863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aca89fb-bb81-4def-baa8-331d669e6694 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.652259 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b52643c-cce6-4ede-9034-3607160ea608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.653026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b52643c-cce6-4ede-9034-3607160ea608 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.653372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b52643c-cce6-4ede-9034-3607160ea608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.653655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b52643c-cce6-4ede-9034-3607160ea608 admin admin] PATH_INFO: `/v3/projects/c5003b0655b940a095ec3534f7ddf884/users/1bd0450b3b264b8facc309e2cb3c683a/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.670751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b52643c-cce6-4ede-9034-3607160ea608 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5003b0655b940a095ec3534f7ddf884, user_id=1bd0450b3b264b8facc309e2cb3c683a, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.679060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b52643c-cce6-4ede-9034-3607160ea608 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.694999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1333/2668] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:45 2022] PUT /identity/v3/projects/c5003b0655b940a095ec3534f7ddf884/users/1bd0450b3b264b8facc309e2cb3c683a/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:45.708271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9e266-5dd0-4695-a869-186a992706a2 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.708703 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9e266-5dd0-4695-a869-186a992706a2 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.709127 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9e266-5dd0-4695-a869-186a992706a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.742894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d5d9e266-5dd0-4695-a869-186a992706a2 None None] MFA Rules not processed for user `1bd0450b3b264b8facc309e2cb3c683a`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:45.744131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d5d9e266-5dd0-4695-a869-186a992706a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:45.757132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:45.833468 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1336/2669] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:45.837960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b52643c-cce6-4ede-9034-3607160ea608 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.848444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:45.912275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b52643c-cce6-4ede-9034-3607160ea608 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.917112 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce63e676-8666-4095-ab2c-9c40bdb53e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:45.917728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce63e676-8666-4095-ab2c-9c40bdb53e71 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:45.918004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce63e676-8666-4095-ab2c-9c40bdb53e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:45.918297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce63e676-8666-4095-ab2c-9c40bdb53e71 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:45.919025 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce63e676-8666-4095-ab2c-9c40bdb53e71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:45.921002 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce63e676-8666-4095-ab2c-9c40bdb53e71 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:45.957246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1334/2670] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:45 2022] POST /identity/v3/projects => generated 379 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:45.962331 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d9e266-5dd0-4695-a869-186a992706a2 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.975099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d9e266-5dd0-4695-a869-186a992706a2 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:45.978559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce63e676-8666-4095-ab2c-9c40bdb53e71 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:45.980415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ce63e676-8666-4095-ab2c-9c40bdb53e71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:46.050848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5020df-d9cb-437b-80a3-7151fe085ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.051995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5020df-d9cb-437b-80a3-7151fe085ba7 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.052439 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5020df-d9cb-437b-80a3-7151fe085ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.052953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5020df-d9cb-437b-80a3-7151fe085ba7 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.054129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5020df-d9cb-437b-80a3-7151fe085ba7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.057513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5020df-d9cb-437b-80a3-7151fe085ba7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.066780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce63e676-8666-4095-ab2c-9c40bdb53e71 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.071574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.072465 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.072786 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.073101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] PATH_INFO: `/v3/projects/92c56a4dfb4f41ce8a43082cb9451e30` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.074400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=92c56a4dfb4f41ce8a43082cb9451e30)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.076825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.086149 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1337/2671] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:45 2022] POST /identity/v3/users => generated 290 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:46.087217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5020df-d9cb-437b-80a3-7151fe085ba7 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.097307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5020df-d9cb-437b-80a3-7151fe085ba7 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.101773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd051c4-a7a0-4b9b-895f-5069faccf43c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.102518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd051c4-a7a0-4b9b-895f-5069faccf43c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.102793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd051c4-a7a0-4b9b-895f-5069faccf43c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.103061 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd051c4-a7a0-4b9b-895f-5069faccf43c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.103690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd051c4-a7a0-4b9b-895f-5069faccf43c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.105272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd051c4-a7a0-4b9b-895f-5069faccf43c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.116000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1338/2672] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:46.120066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd051c4-a7a0-4b9b-895f-5069faccf43c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.129303 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92c56a4dfb4f41ce8a43082cb9451e30', 'request_id': 'req-7d6bba10-6cda-477a-b792-b6ee446bc3b8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.130060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd051c4-a7a0-4b9b-895f-5069faccf43c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.135699 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92c56a4dfb4f41ce8a43082cb9451e30', 'request_id': 'req-7d6bba10-6cda-477a-b792-b6ee446bc3b8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.144852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92c56a4dfb4f41ce8a43082cb9451e30', 'request_id': 'req-7d6bba10-6cda-477a-b792-b6ee446bc3b8'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.157042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] The token cache is being invalidate because project 92c56a4dfb4f41ce8a43082cb9451e30 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:46.157469 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.158286 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.160102 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1335/2673] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:45 2022] DELETE /identity/v3/projects/92c56a4dfb4f41ce8a43082cb9451e30 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:46.164094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.165470 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:46.190236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7cf9db-2cce-4721-a75f-6f58a06ead88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.190847 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7cf9db-2cce-4721-a75f-6f58a06ead88 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.191128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7cf9db-2cce-4721-a75f-6f58a06ead88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.191406 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7cf9db-2cce-4721-a75f-6f58a06ead88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.192057 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7cf9db-2cce-4721-a75f-6f58a06ead88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.193154 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:46.193680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7cf9db-2cce-4721-a75f-6f58a06ead88 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.204589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1339/2674] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:46 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:46.208691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7cf9db-2cce-4721-a75f-6f58a06ead88 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.210570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3c7cf9db-2cce-4721-a75f-6f58a06ead88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:46.222978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7cf9db-2cce-4721-a75f-6f58a06ead88 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.227784 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c848869-418f-4d6e-9532-9e986bc1d1c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.228844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c848869-418f-4d6e-9532-9e986bc1d1c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.229332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c848869-418f-4d6e-9532-9e986bc1d1c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.229812 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c848869-418f-4d6e-9532-9e986bc1d1c4 admin admin] PATH_INFO: `/v3/projects/9602b416db1d42659d01ebbc76fe1ac9/users/ed3b6db32a0848fda9946bc44130ab3e/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.234489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6bba10-6cda-477a-b792-b6ee446bc3b8 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.240152 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.240836 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.241262 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.241538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] PATH_INFO: `/v3/users/6d5680b71f0f42e0bf6e871b9bd76e34` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.243176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d5680b71f0f42e0bf6e871b9bd76e34)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.245416 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.251403 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c848869-418f-4d6e-9532-9e986bc1d1c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9602b416db1d42659d01ebbc76fe1ac9, user_id=ed3b6db32a0848fda9946bc44130ab3e, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.257007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c848869-418f-4d6e-9532-9e986bc1d1c4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.273284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1340/2675] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:46 2022] PUT /identity/v3/projects/9602b416db1d42659d01ebbc76fe1ac9/users/ed3b6db32a0848fda9946bc44130ab3e/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:40:46.284523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7937a38a-9eb2-4f9b-88a8-cdc23299704c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.284793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7937a38a-9eb2-4f9b-88a8-cdc23299704c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.285060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7937a38a-9eb2-4f9b-88a8-cdc23299704c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.293730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:46.297790 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d5680b71f0f42e0bf6e871b9bd76e34'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.298495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d5680b71f0f42e0bf6e871b9bd76e34'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.306368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d5680b71f0f42e0bf6e871b9bd76e34'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.307346 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d5680b71f0f42e0bf6e871b9bd76e34'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.315718 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7937a38a-9eb2-4f9b-88a8-cdc23299704c None None] MFA Rules not processed for user `ed3b6db32a0848fda9946bc44130ab3e`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:46.316895 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d5680b71f0f42e0bf6e871b9bd76e34'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.317051 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7937a38a-9eb2-4f9b-88a8-cdc23299704c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:46.337011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1336/2676] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:46 2022] DELETE /identity/v3/users/6d5680b71f0f42e0bf6e871b9bd76e34 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:40:46.383192 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1341/2677] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:46 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:46.387676 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.389472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:46.456961 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f12c7a-4223-42e5-8059-307421105747 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.462078 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe5813e-dc9e-4f32-a194-f215467f44ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.462724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe5813e-dc9e-4f32-a194-f215467f44ba admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.462991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe5813e-dc9e-4f32-a194-f215467f44ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.463264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe5813e-dc9e-4f32-a194-f215467f44ba admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.463907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe5813e-dc9e-4f32-a194-f215467f44ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.465844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe5813e-dc9e-4f32-a194-f215467f44ba admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.498228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1337/2678] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:46 2022] POST /identity/v3/projects => generated 379 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:46.502487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7937a38a-9eb2-4f9b-88a8-cdc23299704c None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.566884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7937a38a-9eb2-4f9b-88a8-cdc23299704c None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.617396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe5813e-dc9e-4f32-a194-f215467f44ba admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.619408 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-bfe5813e-dc9e-4f32-a194-f215467f44ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:46.643381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd85438-dedf-458c-a073-0a5e441dfb52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.644040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd85438-dedf-458c-a073-0a5e441dfb52 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.644320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd85438-dedf-458c-a073-0a5e441dfb52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.644591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd85438-dedf-458c-a073-0a5e441dfb52 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.645253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd85438-dedf-458c-a073-0a5e441dfb52 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.647261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd85438-dedf-458c-a073-0a5e441dfb52 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.658715 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:46.673330 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1342/2679] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:46 2022] POST /identity/v3/users => generated 296 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:40:46.677069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd85438-dedf-458c-a073-0a5e441dfb52 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.687159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd85438-dedf-458c-a073-0a5e441dfb52 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.691942 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0136125e-d7b0-4a05-b581-27c2a5a7b81e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.692548 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0136125e-d7b0-4a05-b581-27c2a5a7b81e admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.692822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0136125e-d7b0-4a05-b581-27c2a5a7b81e admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.693093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0136125e-d7b0-4a05-b581-27c2a5a7b81e admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.693742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0136125e-d7b0-4a05-b581-27c2a5a7b81e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.695379 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0136125e-d7b0-4a05-b581-27c2a5a7b81e admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.697138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe5813e-dc9e-4f32-a194-f215467f44ba admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.702183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.702816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.703095 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.703367 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] PATH_INFO: `/v3/projects/cb6b544858c649acbf0c7292840b60e5` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.704521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb6b544858c649acbf0c7292840b60e5)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.705905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1343/2680] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:46.706467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.710134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0136125e-d7b0-4a05-b581-27c2a5a7b81e admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.720187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0136125e-d7b0-4a05-b581-27c2a5a7b81e admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.725841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa5f823-3660-4916-bd2f-abac4546d8bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.726525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa5f823-3660-4916-bd2f-abac4546d8bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.726800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa5f823-3660-4916-bd2f-abac4546d8bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.727068 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa5f823-3660-4916-bd2f-abac4546d8bd admin admin] PATH_INFO: `/v3/projects/6cdab7eb6c3c48d38a9544bf49d4a6e9/users/a41467e4f7344bc097ff0b8acab3de84/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.742940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa5f823-3660-4916-bd2f-abac4546d8bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=6cdab7eb6c3c48d38a9544bf49d4a6e9, user_id=a41467e4f7344bc097ff0b8acab3de84, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.745616 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa5f823-3660-4916-bd2f-abac4546d8bd admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.749065 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:46.757878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb6b544858c649acbf0c7292840b60e5', 'request_id': 'req-69c4b426-5375-48da-b12e-eb4dcef66bfe'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.760537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1344/2681] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:46 2022] PUT /identity/v3/projects/6cdab7eb6c3c48d38a9544bf49d4a6e9/users/a41467e4f7344bc097ff0b8acab3de84/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:46.763855 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb6b544858c649acbf0c7292840b60e5', 'request_id': 'req-69c4b426-5375-48da-b12e-eb4dcef66bfe'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.764597 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa5f823-3660-4916-bd2f-abac4546d8bd admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.773472 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb6b544858c649acbf0c7292840b60e5', 'request_id': 'req-69c4b426-5375-48da-b12e-eb4dcef66bfe'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.785553 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] The token cache is being invalidate because project cb6b544858c649acbf0c7292840b60e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:46.785970 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.786752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-69c4b426-5375-48da-b12e-eb4dcef66bfe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:46.788402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1338/2682] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:46 2022] DELETE /identity/v3/projects/cb6b544858c649acbf0c7292840b60e5 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:40:46.793456 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603d2014-3994-4730-a30f-95cbe9108623 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.793706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603d2014-3994-4730-a30f-95cbe9108623 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.793984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603d2014-3994-4730-a30f-95cbe9108623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.815121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-603d2014-3994-4730-a30f-95cbe9108623 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:46.816337 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-603d2014-3994-4730-a30f-95cbe9108623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:46.828803 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa5f823-3660-4916-bd2f-abac4546d8bd admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.833594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46e9f5f-bde3-46c0-b4e4-aa2ff504a171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.834315 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46e9f5f-bde3-46c0-b4e4-aa2ff504a171 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.834660 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46e9f5f-bde3-46c0-b4e4-aa2ff504a171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.834993 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46e9f5f-bde3-46c0-b4e4-aa2ff504a171 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.835706 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46e9f5f-bde3-46c0-b4e4-aa2ff504a171 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.837359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46e9f5f-bde3-46c0-b4e4-aa2ff504a171 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.847665 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1345/2683] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:46 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:46.851571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46e9f5f-bde3-46c0-b4e4-aa2ff504a171 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.853207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-c46e9f5f-bde3-46c0-b4e4-aa2ff504a171 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:46.866034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46e9f5f-bde3-46c0-b4e4-aa2ff504a171 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.871191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991d07a9-aa08-49b7-a923-efa4d69172fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.871859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991d07a9-aa08-49b7-a923-efa4d69172fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.872140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991d07a9-aa08-49b7-a923-efa4d69172fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.872414 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991d07a9-aa08-49b7-a923-efa4d69172fc admin admin] PATH_INFO: `/v3/projects/6cdab7eb6c3c48d38a9544bf49d4a6e9/users/a41467e4f7344bc097ff0b8acab3de84/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.874453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991d07a9-aa08-49b7-a923-efa4d69172fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=6cdab7eb6c3c48d38a9544bf49d4a6e9, user_id=a41467e4f7344bc097ff0b8acab3de84, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.877097 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991d07a9-aa08-49b7-a923-efa4d69172fc admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:46.877211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1339/2684] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:46.885912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ae472f-6b9f-43a6-a7ba-46f72391e07b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.886222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ae472f-6b9f-43a6-a7ba-46f72391e07b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.886492 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ae472f-6b9f-43a6-a7ba-46f72391e07b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.889735 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1346/2685] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:46 2022] PUT /identity/v3/projects/6cdab7eb6c3c48d38a9544bf49d4a6e9/users/a41467e4f7344bc097ff0b8acab3de84/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:46.900755 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1030d60-4c3a-49f8-ad45-b3c70460cf06 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.901038 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1030d60-4c3a-49f8-ad45-b3c70460cf06 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.901305 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1030d60-4c3a-49f8-ad45-b3c70460cf06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.910590 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-51ae472f-6b9f-43a6-a7ba-46f72391e07b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:46.911819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-51ae472f-6b9f-43a6-a7ba-46f72391e07b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:46.929555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d1030d60-4c3a-49f8-ad45-b3c70460cf06 None None] MFA Rules not processed for user `a41467e4f7344bc097ff0b8acab3de84`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:46.930776 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d1030d60-4c3a-49f8-ad45-b3c70460cf06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:46.965690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:46.978555 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1340/2686] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:46.979642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ae472f-6b9f-43a6-a7ba-46f72391e07b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:46.989017 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ae472f-6b9f-43a6-a7ba-46f72391e07b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:46.993578 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9f581b-d1ac-4146-86c7-c0f39777ba81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:46.994199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9f581b-d1ac-4146-86c7-c0f39777ba81 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:46.994479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9f581b-d1ac-4146-86c7-c0f39777ba81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:46.994752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9f581b-d1ac-4146-86c7-c0f39777ba81 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:46.995397 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9f581b-d1ac-4146-86c7-c0f39777ba81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:40:46.995702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9f581b-d1ac-4146-86c7-c0f39777ba81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:46.997360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9f581b-d1ac-4146-86c7-c0f39777ba81 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:47.000868 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1347/2687] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:46 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:40:47.009772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1341/2688] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:40:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:40:47.012340 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1030d60-4c3a-49f8-ad45-b3c70460cf06 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:47.013585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9f581b-d1ac-4146-86c7-c0f39777ba81 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:47.014002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d1030d60-4c3a-49f8-ad45-b3c70460cf06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:47.023933 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9f581b-d1ac-4146-86c7-c0f39777ba81 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:47.028428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79b0af2-02f0-4b4f-b6f7-81e905adb9a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:47.029007 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79b0af2-02f0-4b4f-b6f7-81e905adb9a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:47.029283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79b0af2-02f0-4b4f-b6f7-81e905adb9a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:47.029621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79b0af2-02f0-4b4f-b6f7-81e905adb9a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:47.030257 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79b0af2-02f0-4b4f-b6f7-81e905adb9a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:47.032158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79b0af2-02f0-4b4f-b6f7-81e905adb9a3 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:47.060415 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1342/2689] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:47 2022] POST /identity/v3/projects => generated 391 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:40:47.064857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79b0af2-02f0-4b4f-b6f7-81e905adb9a3 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:47.073472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1030d60-4c3a-49f8-ad45-b3c70460cf06 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:47.090658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:47.123529 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07f4788-0081-4bae-9d02-90aa30ae02bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:47.124398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07f4788-0081-4bae-9d02-90aa30ae02bc service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:47.124670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07f4788-0081-4bae-9d02-90aa30ae02bc service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:47.124929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07f4788-0081-4bae-9d02-90aa30ae02bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:47.128848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79b0af2-02f0-4b4f-b6f7-81e905adb9a3 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:47.133476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ba7cbd-686f-4da8-afa8-4d3629c2be50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:47.134087 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ba7cbd-686f-4da8-afa8-4d3629c2be50 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:47.134400 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ba7cbd-686f-4da8-afa8-4d3629c2be50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:47.134675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ba7cbd-686f-4da8-afa8-4d3629c2be50 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:47.135304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ba7cbd-686f-4da8-afa8-4d3629c2be50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:47.137246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ba7cbd-686f-4da8-afa8-4d3629c2be50 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:47.164644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1343/2690] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:47 2022] POST /identity/v3/users => generated 296 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:47.165656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ba7cbd-686f-4da8-afa8-4d3629c2be50 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:47.176138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ba7cbd-686f-4da8-afa8-4d3629c2be50 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:47.180695 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb82af10-e8b3-4060-b06f-110e58f4d1c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:47.181287 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb82af10-e8b3-4060-b06f-110e58f4d1c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:47.181600 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb82af10-e8b3-4060-b06f-110e58f4d1c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:47.181876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb82af10-e8b3-4060-b06f-110e58f4d1c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:47.182653 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb82af10-e8b3-4060-b06f-110e58f4d1c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:47.184225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb82af10-e8b3-4060-b06f-110e58f4d1c6 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:47.188516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07f4788-0081-4bae-9d02-90aa30ae02bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:47.189967 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07f4788-0081-4bae-9d02-90aa30ae02bc service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:47.194720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1344/2691] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:47 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:47.198625 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb82af10-e8b3-4060-b06f-110e58f4d1c6 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:47.199839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1348/2692] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:47 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:47.209052 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb82af10-e8b3-4060-b06f-110e58f4d1c6 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:47.213937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2658a6f-d549-42b9-abc5-4dbb1c579811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:47.214330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2658a6f-d549-42b9-abc5-4dbb1c579811 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:47.214603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2658a6f-d549-42b9-abc5-4dbb1c579811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:47.214866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2658a6f-d549-42b9-abc5-4dbb1c579811 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:47.215529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2658a6f-d549-42b9-abc5-4dbb1c579811 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:47.217227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2658a6f-d549-42b9-abc5-4dbb1c579811 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:47.228718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1345/2693] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:40:47.234269 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07f4788-0081-4bae-9d02-90aa30ae02bc service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:47.242727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2658a6f-d549-42b9-abc5-4dbb1c579811 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:47.244628 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e2658a6f-d549-42b9-abc5-4dbb1c579811 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:47.245138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07f4788-0081-4bae-9d02-90aa30ae02bc service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:47.249856 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf6b12-d872-4fce-a043-2c07fb949333 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:47.250551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf6b12-d872-4fce-a043-2c07fb949333 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:47.250833 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf6b12-d872-4fce-a043-2c07fb949333 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:47.251105 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf6b12-d872-4fce-a043-2c07fb949333 admin admin] PATH_INFO: `/v3/projects/dc27b46cce614ec49a3b644e58a15175/users/95cc9ed536034a62b22685276e99035f/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:47.257471 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2658a6f-d549-42b9-abc5-4dbb1c579811 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:47.261591 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60149730-23ac-46bf-9d75-7d60e3873a58 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:47.262221 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60149730-23ac-46bf-9d75-7d60e3873a58 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:47.262617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60149730-23ac-46bf-9d75-7d60e3873a58 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:47.262888 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60149730-23ac-46bf-9d75-7d60e3873a58 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:47.265081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-60149730-23ac-46bf-9d75-7d60e3873a58 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:47.266341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcf6b12-d872-4fce-a043-2c07fb949333 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc27b46cce614ec49a3b644e58a15175, user_id=95cc9ed536034a62b22685276e99035f, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:47.268978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcf6b12-d872-4fce-a043-2c07fb949333 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:47.284162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1349/2694] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:47 2022] PUT /identity/v3/projects/dc27b46cce614ec49a3b644e58a15175/users/95cc9ed536034a62b22685276e99035f/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:47.340503 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60149730-23ac-46bf-9d75-7d60e3873a58 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:47.341879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60149730-23ac-46bf-9d75-7d60e3873a58 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:47.352542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1346/2695] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:47 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:47.354661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded1fdf0-1811-49ed-a58f-d7c3eb1deeda None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:47.354948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded1fdf0-1811-49ed-a58f-d7c3eb1deeda None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:47.355220 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded1fdf0-1811-49ed-a58f-d7c3eb1deeda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:47.386644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-ded1fdf0-1811-49ed-a58f-d7c3eb1deeda None None] MFA Rules not processed for user `95cc9ed536034a62b22685276e99035f`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:47.387896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ded1fdf0-1811-49ed-a58f-d7c3eb1deeda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:47.458555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1350/2696] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:47 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:47.470912 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60149730-23ac-46bf-9d75-7d60e3873a58 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:47.480188 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60149730-23ac-46bf-9d75-7d60e3873a58 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:47.513150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:47.535710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e357fe-f527-4f37-9bda-fd9757f33684 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:47.536323 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e357fe-f527-4f37-9bda-fd9757f33684 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:47.536597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e357fe-f527-4f37-9bda-fd9757f33684 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:47.536867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e357fe-f527-4f37-9bda-fd9757f33684 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:47.546216 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e357fe-f527-4f37-9bda-fd9757f33684 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:47.547872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e357fe-f527-4f37-9bda-fd9757f33684 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:47.557721 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1347/2697] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:47 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:53.474267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded1fdf0-1811-49ed-a58f-d7c3eb1deeda None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:53.513110 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:53.542622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded1fdf0-1811-49ed-a58f-d7c3eb1deeda None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:53.548436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:53.549078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:53.549353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:53.549621 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] PATH_INFO: `/v3/users/1bd0450b3b264b8facc309e2cb3c683a` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:53.551000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1bd0450b3b264b8facc309e2cb3c683a)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:53.553099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:53.606856 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1bd0450b3b264b8facc309e2cb3c683a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:53.616435 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1bd0450b3b264b8facc309e2cb3c683a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:53.617143 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1bd0450b3b264b8facc309e2cb3c683a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:53.617864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1bd0450b3b264b8facc309e2cb3c683a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:53.625586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1bd0450b3b264b8facc309e2cb3c683a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:53.649300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1351/2698] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:53 2022] DELETE /identity/v3/users/1bd0450b3b264b8facc309e2cb3c683a => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:53.654196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a46943c-0334-4727-a2c6-720cb1ac3e35 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:53.654671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a46943c-0334-4727-a2c6-720cb1ac3e35 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:53.654792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a46943c-0334-4727-a2c6-720cb1ac3e35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:53.674879 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-4a46943c-0334-4727-a2c6-720cb1ac3e35 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:53.676123 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4a46943c-0334-4727-a2c6-720cb1ac3e35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:53.740443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1348/2699] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:53.750689 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:53.752217 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:53.811793 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66054b43-7814-40a3-820f-3ee44f664d58 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:53.815662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1794e373-656c-48a2-b80a-881f84ea1849 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:53.816263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1794e373-656c-48a2-b80a-881f84ea1849 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:53.816537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1794e373-656c-48a2-b80a-881f84ea1849 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:53.816850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1794e373-656c-48a2-b80a-881f84ea1849 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:53.826095 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1794e373-656c-48a2-b80a-881f84ea1849 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:53.827543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1794e373-656c-48a2-b80a-881f84ea1849 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:53.837365 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1352/2700] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:54.117909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a46943c-0334-4727-a2c6-720cb1ac3e35 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:54.119546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4a46943c-0334-4727-a2c6-720cb1ac3e35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:54.131873 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a46943c-0334-4727-a2c6-720cb1ac3e35 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:54.137288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:54.138029 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:54.138390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:54.138718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] PATH_INFO: `/v3/projects/c5003b0655b940a095ec3534f7ddf884` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:54.139977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5003b0655b940a095ec3534f7ddf884)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:54.142932 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:54.195376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5003b0655b940a095ec3534f7ddf884', 'request_id': 'req-b7d67208-a9d1-4cc6-aada-40713b33ef05'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.201425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5003b0655b940a095ec3534f7ddf884', 'request_id': 'req-b7d67208-a9d1-4cc6-aada-40713b33ef05'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.211006 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5003b0655b940a095ec3534f7ddf884', 'request_id': 'req-b7d67208-a9d1-4cc6-aada-40713b33ef05'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.223308 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] The token cache is being invalidate because project c5003b0655b940a095ec3534f7ddf884 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:54.223588 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.224283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.225857 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1349/2701] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:54 2022] DELETE /identity/v3/projects/c5003b0655b940a095ec3534f7ddf884 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:54.230266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1794e373-656c-48a2-b80a-881f84ea1849 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:54.240027 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1794e373-656c-48a2-b80a-881f84ea1849 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:54.292427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:54.298748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:54.299505 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:54.299936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:54.300351 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] PATH_INFO: `/v3/users/ed3b6db32a0848fda9946bc44130ab3e` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:54.301969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed3b6db32a0848fda9946bc44130ab3e)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:54.304103 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:54.354863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed3b6db32a0848fda9946bc44130ab3e'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.369643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed3b6db32a0848fda9946bc44130ab3e'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.370325 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed3b6db32a0848fda9946bc44130ab3e'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.370976 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed3b6db32a0848fda9946bc44130ab3e'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.378293 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed3b6db32a0848fda9946bc44130ab3e'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.395846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1353/2702] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:54 2022] DELETE /identity/v3/users/ed3b6db32a0848fda9946bc44130ab3e => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:54.683818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:54.685305 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:54.700698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d67208-a9d1-4cc6-aada-40713b33ef05 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:54.771289 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:54.778867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:54.779529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:54.779848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:54.780129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] PATH_INFO: `/v3/projects/9602b416db1d42659d01ebbc76fe1ac9` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:54.781372 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9602b416db1d42659d01ebbc76fe1ac9)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:54.783964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:54.835580 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9602b416db1d42659d01ebbc76fe1ac9', 'request_id': 'req-560283ad-cc24-445b-b8a0-8a7912ad3b50'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.842645 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9602b416db1d42659d01ebbc76fe1ac9', 'request_id': 'req-560283ad-cc24-445b-b8a0-8a7912ad3b50'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.854452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9602b416db1d42659d01ebbc76fe1ac9', 'request_id': 'req-560283ad-cc24-445b-b8a0-8a7912ad3b50'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.865792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] The token cache is being invalidate because project 9602b416db1d42659d01ebbc76fe1ac9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:54.866072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.866827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:54.868335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1350/2703] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:54 2022] DELETE /identity/v3/projects/9602b416db1d42659d01ebbc76fe1ac9 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:54.872535 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:54.874039 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:54.887045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb2b16e-3c22-4a92-ae6a-e8026c88aa97 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:54.947398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:54.948024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:54.948267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:54.948531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] PATH_INFO: `/v3/users/a41467e4f7344bc097ff0b8acab3de84` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:54.949701 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a41467e4f7344bc097ff0b8acab3de84)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:54.951655 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:55.011932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a41467e4f7344bc097ff0b8acab3de84'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:55.020628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a41467e4f7344bc097ff0b8acab3de84'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:55.021304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a41467e4f7344bc097ff0b8acab3de84'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:55.022067 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a41467e4f7344bc097ff0b8acab3de84'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:55.029702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-43a8caa2-d273-4186-889b-cf3ef32a61d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a41467e4f7344bc097ff0b8acab3de84'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:55.041111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:55.048358 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1354/2704] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:54 2022] DELETE /identity/v3/users/a41467e4f7344bc097ff0b8acab3de84 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:55.359481 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:55.361306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:55.374901 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560283ad-cc24-445b-b8a0-8a7912ad3b50 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:55.432606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:55.433151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:55.433425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:55.433675 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] PATH_INFO: `/v3/projects/6cdab7eb6c3c48d38a9544bf49d4a6e9` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:55.434812 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=6cdab7eb6c3c48d38a9544bf49d4a6e9)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:55.436724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:55.486294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6cdab7eb6c3c48d38a9544bf49d4a6e9', 'request_id': 'req-e787530f-ca17-4e93-a4d8-6d99944239bf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:55.493296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6cdab7eb6c3c48d38a9544bf49d4a6e9', 'request_id': 'req-e787530f-ca17-4e93-a4d8-6d99944239bf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:55.503717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6cdab7eb6c3c48d38a9544bf49d4a6e9', 'request_id': 'req-e787530f-ca17-4e93-a4d8-6d99944239bf'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:55.516614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] The token cache is being invalidate because project 6cdab7eb6c3c48d38a9544bf49d4a6e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:55.516906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:55.517642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e787530f-ca17-4e93-a4d8-6d99944239bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:55.519318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1351/2705] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:55 2022] DELETE /identity/v3/projects/6cdab7eb6c3c48d38a9544bf49d4a6e9 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:55.527155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad2bac-1f89-4b2f-bcc4-348fbd4a678e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:55.527421 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad2bac-1f89-4b2f-bcc4-348fbd4a678e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:55.527691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad2bac-1f89-4b2f-bcc4-348fbd4a678e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:55.547609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-e9ad2bac-1f89-4b2f-bcc4-348fbd4a678e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:55.548795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e9ad2bac-1f89-4b2f-bcc4-348fbd4a678e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:55.615471 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1355/2706] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:55.623906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41432cd-f4c8-40f9-aa9f-afb21f33a353 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:55.624190 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41432cd-f4c8-40f9-aa9f-afb21f33a353 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:55.624462 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41432cd-f4c8-40f9-aa9f-afb21f33a353 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:55.645832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d41432cd-f4c8-40f9-aa9f-afb21f33a353 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:55.647170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d41432cd-f4c8-40f9-aa9f-afb21f33a353 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:55.655135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1352/2707] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:55.659502 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ad2bac-1f89-4b2f-bcc4-348fbd4a678e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:55.669560 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ad2bac-1f89-4b2f-bcc4-348fbd4a678e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:55.674975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ab22bd-56d5-457d-8260-317d2e3bde03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:55.675581 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ab22bd-56d5-457d-8260-317d2e3bde03 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:55.675855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ab22bd-56d5-457d-8260-317d2e3bde03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:55.676124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ab22bd-56d5-457d-8260-317d2e3bde03 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:55.676907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ab22bd-56d5-457d-8260-317d2e3bde03 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:40:55.677070 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ab22bd-56d5-457d-8260-317d2e3bde03 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:55.678722 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ab22bd-56d5-457d-8260-317d2e3bde03 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:55.690384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1356/2708] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:40:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:40:55.695512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41432cd-f4c8-40f9-aa9f-afb21f33a353 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:55.706150 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41432cd-f4c8-40f9-aa9f-afb21f33a353 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:55.711334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb2389f-942c-4186-aa69-4bef52bc40e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:55.711939 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb2389f-942c-4186-aa69-4bef52bc40e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:55.712222 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb2389f-942c-4186-aa69-4bef52bc40e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:55.712495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb2389f-942c-4186-aa69-4bef52bc40e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:55.713158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb2389f-942c-4186-aa69-4bef52bc40e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:55.715198 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb2389f-942c-4186-aa69-4bef52bc40e4 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:55.725061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:55.752278 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1353/2709] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:55 2022] POST /identity/v3/projects => generated 371 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:55.756850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ab22bd-56d5-457d-8260-317d2e3bde03 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:55.767575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ab22bd-56d5-457d-8260-317d2e3bde03 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:55.827642 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae83c87-a427-410f-b84c-b15d26e1b46b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:55.828818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae83c87-a427-410f-b84c-b15d26e1b46b admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:55.829181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae83c87-a427-410f-b84c-b15d26e1b46b admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:55.829524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae83c87-a427-410f-b84c-b15d26e1b46b admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:55.830465 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae83c87-a427-410f-b84c-b15d26e1b46b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:55.833396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae83c87-a427-410f-b84c-b15d26e1b46b admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:55.862080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1357/2710] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:55 2022] POST /identity/v3/users => generated 286 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:55.866421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb2389f-942c-4186-aa69-4bef52bc40e4 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:55.876674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb2389f-942c-4186-aa69-4bef52bc40e4 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:55.882116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a05ea3-d113-4a7f-b478-a04b71ae7335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:55.882867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a05ea3-d113-4a7f-b478-a04b71ae7335 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:55.883138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a05ea3-d113-4a7f-b478-a04b71ae7335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:55.883425 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a05ea3-d113-4a7f-b478-a04b71ae7335 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:55.884065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a05ea3-d113-4a7f-b478-a04b71ae7335 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:55.885683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a05ea3-d113-4a7f-b478-a04b71ae7335 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:55.896068 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1354/2711] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:55.900167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae83c87-a427-410f-b84c-b15d26e1b46b admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:55.910252 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae83c87-a427-410f-b84c-b15d26e1b46b admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:55.915291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5888513-d956-457d-ad14-18a5e5225fe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:55.915899 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5888513-d956-457d-ad14-18a5e5225fe1 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:55.916152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5888513-d956-457d-ad14-18a5e5225fe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:55.916423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5888513-d956-457d-ad14-18a5e5225fe1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:55.917063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5888513-d956-457d-ad14-18a5e5225fe1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:55.918680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5888513-d956-457d-ad14-18a5e5225fe1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:55.929266 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1358/2712] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:55.933405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a05ea3-d113-4a7f-b478-a04b71ae7335 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:55.944306 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a05ea3-d113-4a7f-b478-a04b71ae7335 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:55.949796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced4ef63-26d8-42bd-83fc-c90031f51ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:55.950470 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced4ef63-26d8-42bd-83fc-c90031f51ce3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:55.950749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced4ef63-26d8-42bd-83fc-c90031f51ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:55.951016 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced4ef63-26d8-42bd-83fc-c90031f51ce3 admin admin] PATH_INFO: `/v3/projects/769a7ea5143a4486bdb631beae2b49cd/users/955954d9035849d9b5fed16a1113200a/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:55.967479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced4ef63-26d8-42bd-83fc-c90031f51ce3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=769a7ea5143a4486bdb631beae2b49cd, user_id=955954d9035849d9b5fed16a1113200a, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:55.970173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced4ef63-26d8-42bd-83fc-c90031f51ce3 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:55.991859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1355/2713] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:55 2022] PUT /identity/v3/projects/769a7ea5143a4486bdb631beae2b49cd/users/955954d9035849d9b5fed16a1113200a/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:56.004360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da595b24-876b-43dc-bc55-29384d78351c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:56.004777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da595b24-876b-43dc-bc55-29384d78351c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:56.005059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da595b24-876b-43dc-bc55-29384d78351c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:56.033082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-da595b24-876b-43dc-bc55-29384d78351c None None] MFA Rules not processed for user `955954d9035849d9b5fed16a1113200a`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:56.034366 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-da595b24-876b-43dc-bc55-29384d78351c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:56.091215 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:56.110302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1359/2714] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:56 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:56.122621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced4ef63-26d8-42bd-83fc-c90031f51ce3 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:56.124709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ced4ef63-26d8-42bd-83fc-c90031f51ce3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:56.182876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced4ef63-26d8-42bd-83fc-c90031f51ce3 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:56.187616 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7b1fd8-6440-4fca-9417-0698bdf80f36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:56.188225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7b1fd8-6440-4fca-9417-0698bdf80f36 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:56.188476 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7b1fd8-6440-4fca-9417-0698bdf80f36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:56.188748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7b1fd8-6440-4fca-9417-0698bdf80f36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:56.199264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7b1fd8-6440-4fca-9417-0698bdf80f36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:56.200687 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7b1fd8-6440-4fca-9417-0698bdf80f36 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:56.211793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1356/2715] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:56 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:56.234875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da595b24-876b-43dc-bc55-29384d78351c None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:56.302414 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da595b24-876b-43dc-bc55-29384d78351c None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:56.308201 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:56.309223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:56.309591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:56.309914 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] PATH_INFO: `/v3/users/955954d9035849d9b5fed16a1113200a` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:56.311688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] RBAC: Authorizing `identity:delete_user(user_id=955954d9035849d9b5fed16a1113200a)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:56.314719 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:56.368987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '955954d9035849d9b5fed16a1113200a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:56.377758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '955954d9035849d9b5fed16a1113200a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:56.378429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '955954d9035849d9b5fed16a1113200a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:56.379081 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '955954d9035849d9b5fed16a1113200a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:56.386680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '955954d9035849d9b5fed16a1113200a'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:56.403284 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1360/2716] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:40:56 2022] DELETE /identity/v3/users/955954d9035849d9b5fed16a1113200a => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:56.407859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb37f384-6f1c-4c9d-9005-d6920a16008b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:56.408139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb37f384-6f1c-4c9d-9005-d6920a16008b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:56.408406 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb37f384-6f1c-4c9d-9005-d6920a16008b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:56.427393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-cb37f384-6f1c-4c9d-9005-d6920a16008b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:56.428651 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-cb37f384-6f1c-4c9d-9005-d6920a16008b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:56.489581 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1357/2717] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:56.499999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:56.501516 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:56.564208 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f7384d-fb15-417b-b509-dd4c2e22579d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:56.568354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570f8267-71a8-45bc-a099-2c5034abf291 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:56.569011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570f8267-71a8-45bc-a099-2c5034abf291 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:56.569307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570f8267-71a8-45bc-a099-2c5034abf291 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:56.569573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570f8267-71a8-45bc-a099-2c5034abf291 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:56.580022 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570f8267-71a8-45bc-a099-2c5034abf291 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:56.581442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570f8267-71a8-45bc-a099-2c5034abf291 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:56.591895 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1361/2718] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:40:56.876988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb37f384-6f1c-4c9d-9005-d6920a16008b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:56.878535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-cb37f384-6f1c-4c9d-9005-d6920a16008b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:56.891423 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb37f384-6f1c-4c9d-9005-d6920a16008b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:56.896998 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:56.897725 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:56.897999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:56.898335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] PATH_INFO: `/v3/projects/769a7ea5143a4486bdb631beae2b49cd` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:56.899448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] RBAC: Authorizing `identity:delete_project(project_id=769a7ea5143a4486bdb631beae2b49cd)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:56.901424 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:56.953039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '769a7ea5143a4486bdb631beae2b49cd', 'request_id': 'req-1e3b87c1-3c84-4991-a5c1-28470718d775'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:56.955359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:56.960065 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '769a7ea5143a4486bdb631beae2b49cd', 'request_id': 'req-1e3b87c1-3c84-4991-a5c1-28470718d775'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:56.969370 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '769a7ea5143a4486bdb631beae2b49cd', 'request_id': 'req-1e3b87c1-3c84-4991-a5c1-28470718d775'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:56.981378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] The token cache is being invalidate because project 769a7ea5143a4486bdb631beae2b49cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:40:56.981579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:56.982201 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e3b87c1-3c84-4991-a5c1-28470718d775 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:40:56.983685 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1358/2719] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:40:56 2022] DELETE /identity/v3/projects/769a7ea5143a4486bdb631beae2b49cd => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:56.988744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183e39bf-b495-4051-a128-d319a590170e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:56.989242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183e39bf-b495-4051-a128-d319a590170e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:56.989344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183e39bf-b495-4051-a128-d319a590170e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:57.009207 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-183e39bf-b495-4051-a128-d319a590170e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:57.010359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-183e39bf-b495-4051-a128-d319a590170e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:57.019021 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:40:57.074251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1362/2720] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:57.082597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7f0ac2-2540-4f77-a2d0-d7b31e32b88d None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:57.082880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7f0ac2-2540-4f77-a2d0-d7b31e32b88d None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:57.083151 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7f0ac2-2540-4f77-a2d0-d7b31e32b88d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:57.106284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-5f7f0ac2-2540-4f77-a2d0-d7b31e32b88d None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:57.107509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-5f7f0ac2-2540-4f77-a2d0-d7b31e32b88d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:57.116623 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1359/2721] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:57.122553 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183e39bf-b495-4051-a128-d319a590170e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:57.132746 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183e39bf-b495-4051-a128-d319a590170e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:57.137955 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38243ca-167e-472f-a70d-7909cbe4f79a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:57.138601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38243ca-167e-472f-a70d-7909cbe4f79a admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:57.138900 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38243ca-167e-472f-a70d-7909cbe4f79a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:57.139173 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38243ca-167e-472f-a70d-7909cbe4f79a admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:57.139826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38243ca-167e-472f-a70d-7909cbe4f79a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:40:57.140128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38243ca-167e-472f-a70d-7909cbe4f79a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:57.141744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38243ca-167e-472f-a70d-7909cbe4f79a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:57.154154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1363/2722] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:40:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:40:57.158022 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7f0ac2-2540-4f77-a2d0-d7b31e32b88d None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:57.168277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7f0ac2-2540-4f77-a2d0-d7b31e32b88d None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:57.174210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d710a93-b027-4bbe-931e-802457f4f87e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:57.174809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d710a93-b027-4bbe-931e-802457f4f87e admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:57.175081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d710a93-b027-4bbe-931e-802457f4f87e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:57.175345 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d710a93-b027-4bbe-931e-802457f4f87e admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:57.175942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d710a93-b027-4bbe-931e-802457f4f87e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:57.177864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d710a93-b027-4bbe-931e-802457f4f87e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:57.209073 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1360/2723] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:40:57 2022] POST /identity/v3/projects => generated 377 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:57.213515 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38243ca-167e-472f-a70d-7909cbe4f79a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:57.223493 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38243ca-167e-472f-a70d-7909cbe4f79a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:57.280337 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0dc99e-122c-4760-b352-09f19da1db8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:57.280860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0dc99e-122c-4760-b352-09f19da1db8d admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:57.281119 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0dc99e-122c-4760-b352-09f19da1db8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:57.281384 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0dc99e-122c-4760-b352-09f19da1db8d admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:57.282011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0dc99e-122c-4760-b352-09f19da1db8d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:57.283947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0dc99e-122c-4760-b352-09f19da1db8d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:57.308222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1364/2724] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:40:57 2022] POST /identity/v3/users => generated 289 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:40:57.313911 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d710a93-b027-4bbe-931e-802457f4f87e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:57.325277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d710a93-b027-4bbe-931e-802457f4f87e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:57.330438 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13ebcf4-3f67-460f-9a3d-81af8666f73d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:57.331041 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ebcf4-3f67-460f-9a3d-81af8666f73d admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:57.331313 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ebcf4-3f67-460f-9a3d-81af8666f73d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:57.331575 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13ebcf4-3f67-460f-9a3d-81af8666f73d admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:57.332240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13ebcf4-3f67-460f-9a3d-81af8666f73d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:57.333861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13ebcf4-3f67-460f-9a3d-81af8666f73d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:57.345796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1361/2725] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:57.349792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0dc99e-122c-4760-b352-09f19da1db8d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:57.360860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0dc99e-122c-4760-b352-09f19da1db8d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:57.367078 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9715db-7183-4411-a723-7b102ffd75f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:57.367680 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9715db-7183-4411-a723-7b102ffd75f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:57.367953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9715db-7183-4411-a723-7b102ffd75f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:57.368218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9715db-7183-4411-a723-7b102ffd75f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:57.368844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9715db-7183-4411-a723-7b102ffd75f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:57.370512 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9715db-7183-4411-a723-7b102ffd75f9 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:57.381570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1365/2726] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:40:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:40:57.386279 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13ebcf4-3f67-460f-9a3d-81af8666f73d admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:57.399976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13ebcf4-3f67-460f-9a3d-81af8666f73d admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:57.405450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2aade5-45f0-4608-acdb-64d0abd35875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:57.406145 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2aade5-45f0-4608-acdb-64d0abd35875 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:57.406399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2aade5-45f0-4608-acdb-64d0abd35875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:57.406668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2aade5-45f0-4608-acdb-64d0abd35875 admin admin] PATH_INFO: `/v3/projects/71cf09976dd049f2bca4d2aa649c4a9d/users/0a51c31e8edc4775823b365f75e2349e/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:57.421568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2aade5-45f0-4608-acdb-64d0abd35875 admin admin] RBAC: Authorizing `identity:create_grant(project_id=71cf09976dd049f2bca4d2aa649c4a9d, user_id=0a51c31e8edc4775823b365f75e2349e, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:57.424238 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2aade5-45f0-4608-acdb-64d0abd35875 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:57.439028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1362/2727] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:40:57 2022] PUT /identity/v3/projects/71cf09976dd049f2bca4d2aa649c4a9d/users/0a51c31e8edc4775823b365f75e2349e/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:40:57.450789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6430bb4-9761-406a-a4ad-5847ac119f01 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:57.451076 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6430bb4-9761-406a-a4ad-5847ac119f01 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:57.451362 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6430bb4-9761-406a-a4ad-5847ac119f01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:57.481704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a6430bb4-9761-406a-a4ad-5847ac119f01 None None] MFA Rules not processed for user `0a51c31e8edc4775823b365f75e2349e`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:40:57.482953 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a6430bb4-9761-406a-a4ad-5847ac119f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:57.553649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1366/2728] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:40:57 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:40:57.566231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2aade5-45f0-4608-acdb-64d0abd35875 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:40:57.567946 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7c2aade5-45f0-4608-acdb-64d0abd35875 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:40:57.626615 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2aade5-45f0-4608-acdb-64d0abd35875 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:40:57.630677 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24975db-2a8f-487f-a7e9-8f15fb4f7aeb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ce646dcb8124685ac7a1c1eca664d04', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:40:57.631293 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24975db-2a8f-487f-a7e9-8f15fb4f7aeb service glance] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:40:57.631604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24975db-2a8f-487f-a7e9-8f15fb4f7aeb service glance] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:40:57.631863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24975db-2a8f-487f-a7e9-8f15fb4f7aeb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:40:57.641239 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24975db-2a8f-487f-a7e9-8f15fb4f7aeb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:40:57.642755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24975db-2a8f-487f-a7e9-8f15fb4f7aeb service glance] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:40:57.653593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1363/2729] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:40:57 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:41:06.390401 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6430bb4-9761-406a-a4ad-5847ac119f01 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:06.391960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a6430bb4-9761-406a-a4ad-5847ac119f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:06.403496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:06.455122 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6430bb4-9761-406a-a4ad-5847ac119f01 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:06.458765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8903c5f4-edcd-4553-9a4b-7b181379772a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:06.459391 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8903c5f4-edcd-4553-9a4b-7b181379772a service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:06.459664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8903c5f4-edcd-4553-9a4b-7b181379772a service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:06.459928 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8903c5f4-edcd-4553-9a4b-7b181379772a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:06.461671 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-8903c5f4-edcd-4553-9a4b-7b181379772a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:06.471297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8903c5f4-edcd-4553-9a4b-7b181379772a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:06.472707 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8903c5f4-edcd-4553-9a4b-7b181379772a service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:06.482485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1367/2730] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:07.648177 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=59,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:10.150994 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24975db-2a8f-487f-a7e9-8f15fb4f7aeb service glance] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:10.152667 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e24975db-2a8f-487f-a7e9-8f15fb4f7aeb service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:10.226652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24975db-2a8f-487f-a7e9-8f15fb4f7aeb service glance] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:10.232900 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:10.233744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:10.234039 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:10.234419 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] PATH_INFO: `/v3/users/95cc9ed536034a62b22685276e99035f` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:10.236399 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=95cc9ed536034a62b22685276e99035f)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:10.239054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:10.299203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95cc9ed536034a62b22685276e99035f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:10.299819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95cc9ed536034a62b22685276e99035f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:10.307744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95cc9ed536034a62b22685276e99035f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:10.308522 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95cc9ed536034a62b22685276e99035f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:10.318631 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95cc9ed536034a62b22685276e99035f'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:10.336213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1364/2731] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:10 2022] DELETE /identity/v3/users/95cc9ed536034a62b22685276e99035f => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:10.340946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66bd679-05ad-4c7d-ab8a-6bdd8a3e343c None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:10.341434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66bd679-05ad-4c7d-ab8a-6bdd8a3e343c None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:10.341539 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66bd679-05ad-4c7d-ab8a-6bdd8a3e343c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:10.361841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a66bd679-05ad-4c7d-ab8a-6bdd8a3e343c None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:10.363277 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a66bd679-05ad-4c7d-ab8a-6bdd8a3e343c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:10.431167 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1368/2732] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:10.442024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:10.444557 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:10.517333 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae064cc-e58e-4b3a-9eb7-417ae20e7d5f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:10.521441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0046964-5365-4d0e-a2b8-7dab576e3412 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:10.522281 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0046964-5365-4d0e-a2b8-7dab576e3412 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:10.522554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0046964-5365-4d0e-a2b8-7dab576e3412 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:10.522827 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0046964-5365-4d0e-a2b8-7dab576e3412 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:10.533098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0046964-5365-4d0e-a2b8-7dab576e3412 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:10.534940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0046964-5365-4d0e-a2b8-7dab576e3412 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:10.544387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1365/2733] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:10.841278 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66bd679-05ad-4c7d-ab8a-6bdd8a3e343c None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:10.842843 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a66bd679-05ad-4c7d-ab8a-6bdd8a3e343c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:10.855800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66bd679-05ad-4c7d-ab8a-6bdd8a3e343c None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:10.862314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:10.862977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:10.863467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:10.863627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] PATH_INFO: `/v3/projects/dc27b46cce614ec49a3b644e58a15175` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:10.864752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc27b46cce614ec49a3b644e58a15175)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:10.867062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:10.928652 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc27b46cce614ec49a3b644e58a15175', 'request_id': 'req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:10.934011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc27b46cce614ec49a3b644e58a15175', 'request_id': 'req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:10.942748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc27b46cce614ec49a3b644e58a15175', 'request_id': 'req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:10.957982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] The token cache is being invalidate because project dc27b46cce614ec49a3b644e58a15175 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:10.958481 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:10.959176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:10.960730 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1369/2734] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:10 2022] DELETE /identity/v3/projects/dc27b46cce614ec49a3b644e58a15175 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:17.776592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0046964-5365-4d0e-a2b8-7dab576e3412 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:17.778231 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e0046964-5365-4d0e-a2b8-7dab576e3412 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:17.817062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:17.878210 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0046964-5365-4d0e-a2b8-7dab576e3412 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:17.959304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c49d17-9d58-4a8d-a683-c8b38e6ebf22 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:17.959965 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c49d17-9d58-4a8d-a683-c8b38e6ebf22 service nova] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:17.960317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c49d17-9d58-4a8d-a683-c8b38e6ebf22 service nova] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:17.960949 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c49d17-9d58-4a8d-a683-c8b38e6ebf22 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:17.962984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e9c49d17-9d58-4a8d-a683-c8b38e6ebf22 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:17.978981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c49d17-9d58-4a8d-a683-c8b38e6ebf22 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:17.982566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c49d17-9d58-4a8d-a683-c8b38e6ebf22 service nova] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:17.996451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1366/2735] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:20.953075 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=76,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:23.807667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:23.809154 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:23.870894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7822ea03-9301-4cb8-9fa0-e1a4c1c371d3 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:23.874472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb728510-342d-4be5-93fa-493207590b9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:23.875094 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb728510-342d-4be5-93fa-493207590b9d service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:23.875359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb728510-342d-4be5-93fa-493207590b9d service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:23.875629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb728510-342d-4be5-93fa-493207590b9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:23.877429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-eb728510-342d-4be5-93fa-493207590b9d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:23.949757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb728510-342d-4be5-93fa-493207590b9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:23.951218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb728510-342d-4be5-93fa-493207590b9d service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:23.962976 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1370/2736] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:23.998539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c49d17-9d58-4a8d-a683-c8b38e6ebf22 service nova] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:24.000110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e9c49d17-9d58-4a8d-a683-c8b38e6ebf22 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:24.014509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c49d17-9d58-4a8d-a683-c8b38e6ebf22 service nova] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:24.019828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:24.020467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:24.020751 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:24.021027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] PATH_INFO: `/v3/users/0a51c31e8edc4775823b365f75e2349e` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:24.022203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a51c31e8edc4775823b365f75e2349e)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:24.024389 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:24.074397 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a51c31e8edc4775823b365f75e2349e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:24.075047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a51c31e8edc4775823b365f75e2349e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:24.082548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a51c31e8edc4775823b365f75e2349e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:24.083169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a51c31e8edc4775823b365f75e2349e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:24.092525 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a51c31e8edc4775823b365f75e2349e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:24.111384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1367/2737] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:23 2022] DELETE /identity/v3/users/0a51c31e8edc4775823b365f75e2349e => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:24.115962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6d2f2-f2e2-40d4-82f2-e3e0697368d8 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:24.116251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6d2f2-f2e2-40d4-82f2-e3e0697368d8 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:24.116525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6d2f2-f2e2-40d4-82f2-e3e0697368d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:24.137550 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-37a6d2f2-f2e2-40d4-82f2-e3e0697368d8 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:24.138724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-37a6d2f2-f2e2-40d4-82f2-e3e0697368d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:24.204302 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1371/2738] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:24.233448 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:24.235128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:24.256937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:24.298813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df5536d-e78d-4094-9203-7b499481c349 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:24.303824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439b46e9-2350-4291-81c7-896da1188d34 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:24.304484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439b46e9-2350-4291-81c7-896da1188d34 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:24.304759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439b46e9-2350-4291-81c7-896da1188d34 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:24.305040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439b46e9-2350-4291-81c7-896da1188d34 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:24.315558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439b46e9-2350-4291-81c7-896da1188d34 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:24.317014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439b46e9-2350-4291-81c7-896da1188d34 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:24.326964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1368/2739] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:24.748155 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6d2f2-f2e2-40d4-82f2-e3e0697368d8 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:24.749782 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-37a6d2f2-f2e2-40d4-82f2-e3e0697368d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:24.762537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6d2f2-f2e2-40d4-82f2-e3e0697368d8 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:24.767684 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:24.769084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:24.769449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:24.769783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] PATH_INFO: `/v3/projects/71cf09976dd049f2bca4d2aa649c4a9d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:24.771283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] RBAC: Authorizing `identity:delete_project(project_id=71cf09976dd049f2bca4d2aa649c4a9d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:24.773423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:24.825026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:24.828079 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '71cf09976dd049f2bca4d2aa649c4a9d', 'request_id': 'req-c2c254c3-a3e9-441d-a46e-ffedb70ac756'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:24.834702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '71cf09976dd049f2bca4d2aa649c4a9d', 'request_id': 'req-c2c254c3-a3e9-441d-a46e-ffedb70ac756'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:24.842987 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '71cf09976dd049f2bca4d2aa649c4a9d', 'request_id': 'req-c2c254c3-a3e9-441d-a46e-ffedb70ac756'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:24.854883 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] The token cache is being invalidate because project 71cf09976dd049f2bca4d2aa649c4a9d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:24.855196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:24.855850 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-c2c254c3-a3e9-441d-a46e-ffedb70ac756 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:24.857344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1372/2740] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:24 2022] DELETE /identity/v3/projects/71cf09976dd049f2bca4d2aa649c4a9d => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:24.862702 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3737762-c8c9-4aa8-8b55-68d60a75f51a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:24.863165 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3737762-c8c9-4aa8-8b55-68d60a75f51a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:24.863429 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3737762-c8c9-4aa8-8b55-68d60a75f51a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:24.885758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-d3737762-c8c9-4aa8-8b55-68d60a75f51a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:24.887004 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-d3737762-c8c9-4aa8-8b55-68d60a75f51a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:24.952351 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1369/2741] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:25.024080 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db8f792-503f-465c-bf23-1bf512fa685a None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.024403 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db8f792-503f-465c-bf23-1bf512fa685a None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.024839 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db8f792-503f-465c-bf23-1bf512fa685a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.045907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-2db8f792-503f-465c-bf23-1bf512fa685a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:25.047263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2db8f792-503f-465c-bf23-1bf512fa685a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:25.056148 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1373/2742] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:25.060526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3737762-c8c9-4aa8-8b55-68d60a75f51a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.070543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3737762-c8c9-4aa8-8b55-68d60a75f51a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.075291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4b5ffe-f640-43eb-bb77-d7d7f3b0213c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.075907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4b5ffe-f640-43eb-bb77-d7d7f3b0213c admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.076176 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4b5ffe-f640-43eb-bb77-d7d7f3b0213c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.076445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4b5ffe-f640-43eb-bb77-d7d7f3b0213c admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.077099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4b5ffe-f640-43eb-bb77-d7d7f3b0213c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:41:25.077394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4b5ffe-f640-43eb-bb77-d7d7f3b0213c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.079402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4b5ffe-f640-43eb-bb77-d7d7f3b0213c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.092589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1370/2743] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:41:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 13 15:41:25.096827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db8f792-503f-465c-bf23-1bf512fa685a None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.106503 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db8f792-503f-465c-bf23-1bf512fa685a None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.111303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6002eb-25a0-4db1-83ae-265aec43d4a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.111929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6002eb-25a0-4db1-83ae-265aec43d4a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.112199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6002eb-25a0-4db1-83ae-265aec43d4a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.112464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6002eb-25a0-4db1-83ae-265aec43d4a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.113117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6002eb-25a0-4db1-83ae-265aec43d4a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.115083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6002eb-25a0-4db1-83ae-265aec43d4a8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.155885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1374/2744] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:25 2022] POST /identity/v3/projects => generated 385 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:25.157101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4b5ffe-f640-43eb-bb77-d7d7f3b0213c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.168542 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4b5ffe-f640-43eb-bb77-d7d7f3b0213c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.226146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd072051-0df8-4274-bb4e-9b64bb3b3f64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.226813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd072051-0df8-4274-bb4e-9b64bb3b3f64 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.227101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd072051-0df8-4274-bb4e-9b64bb3b3f64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.228110 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd072051-0df8-4274-bb4e-9b64bb3b3f64 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.228500 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd072051-0df8-4274-bb4e-9b64bb3b3f64 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.230547 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd072051-0df8-4274-bb4e-9b64bb3b3f64 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.253436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1371/2745] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:25 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:25.257588 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6002eb-25a0-4db1-83ae-265aec43d4a8 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.268662 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6002eb-25a0-4db1-83ae-265aec43d4a8 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.274475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0c7601-4abf-4e55-b0ee-455410597196 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.275200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0c7601-4abf-4e55-b0ee-455410597196 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.275480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0c7601-4abf-4e55-b0ee-455410597196 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.275749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0c7601-4abf-4e55-b0ee-455410597196 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.276392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0c7601-4abf-4e55-b0ee-455410597196 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.278010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0c7601-4abf-4e55-b0ee-455410597196 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.288714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1375/2746] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:25.293663 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd072051-0df8-4274-bb4e-9b64bb3b3f64 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.304546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd072051-0df8-4274-bb4e-9b64bb3b3f64 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.310129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa52a52-b629-48bf-b5af-5927cf99a910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.310758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa52a52-b629-48bf-b5af-5927cf99a910 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.311031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa52a52-b629-48bf-b5af-5927cf99a910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.311300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa52a52-b629-48bf-b5af-5927cf99a910 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.311942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa52a52-b629-48bf-b5af-5927cf99a910 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.313559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa52a52-b629-48bf-b5af-5927cf99a910 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.329898 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1372/2747] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:25.331237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0c7601-4abf-4e55-b0ee-455410597196 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.342304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0c7601-4abf-4e55-b0ee-455410597196 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.347997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a38365-7f83-46ad-8b9f-29851392788d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.348659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a38365-7f83-46ad-8b9f-29851392788d admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.349059 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a38365-7f83-46ad-8b9f-29851392788d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.349329 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a38365-7f83-46ad-8b9f-29851392788d admin admin] PATH_INFO: `/v3/projects/2a3041f6bdbf427aa10b3923355f6985/users/b8b58ad122794b0db0443733d342bad8/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.365800 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a38365-7f83-46ad-8b9f-29851392788d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a3041f6bdbf427aa10b3923355f6985, user_id=b8b58ad122794b0db0443733d342bad8, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.368705 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a38365-7f83-46ad-8b9f-29851392788d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.383295 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1376/2748] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:25 2022] PUT /identity/v3/projects/2a3041f6bdbf427aa10b3923355f6985/users/b8b58ad122794b0db0443733d342bad8/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:25.393780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c703cfe5-95b4-4ea8-b512-dc95d351688c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.394149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c703cfe5-95b4-4ea8-b512-dc95d351688c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.394596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c703cfe5-95b4-4ea8-b512-dc95d351688c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.424134 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c703cfe5-95b4-4ea8-b512-dc95d351688c None None] MFA Rules not processed for user `b8b58ad122794b0db0443733d342bad8`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:25.425505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c703cfe5-95b4-4ea8-b512-dc95d351688c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:25.438564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:25.507530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1373/2749] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:25.511808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a38365-7f83-46ad-8b9f-29851392788d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.584467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a38365-7f83-46ad-8b9f-29851392788d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.590432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918acbf9-ad5f-4e3f-9561-03f665958d61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.591054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918acbf9-ad5f-4e3f-9561-03f665958d61 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.591341 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918acbf9-ad5f-4e3f-9561-03f665958d61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.591613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918acbf9-ad5f-4e3f-9561-03f665958d61 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.592282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918acbf9-ad5f-4e3f-9561-03f665958d61 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.594463 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918acbf9-ad5f-4e3f-9561-03f665958d61 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.649559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1377/2750] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:25 2022] POST /identity/v3/projects => generated 385 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:41:25.655868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c703cfe5-95b4-4ea8-b512-dc95d351688c None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.670597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c703cfe5-95b4-4ea8-b512-dc95d351688c None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.738813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c368c3cf-e3be-4820-a68b-f654d9cda04f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.739757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c368c3cf-e3be-4820-a68b-f654d9cda04f admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.739757 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c368c3cf-e3be-4820-a68b-f654d9cda04f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.739980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c368c3cf-e3be-4820-a68b-f654d9cda04f admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.740596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c368c3cf-e3be-4820-a68b-f654d9cda04f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.742578 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c368c3cf-e3be-4820-a68b-f654d9cda04f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.776409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1374/2751] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:25 2022] POST /identity/v3/users => generated 293 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:25.777703 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918acbf9-ad5f-4e3f-9561-03f665958d61 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.788300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918acbf9-ad5f-4e3f-9561-03f665958d61 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.795779 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c126cb09-7e98-4da0-b3b3-4ac812f1e152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.796777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c126cb09-7e98-4da0-b3b3-4ac812f1e152 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.797004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c126cb09-7e98-4da0-b3b3-4ac812f1e152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.797282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c126cb09-7e98-4da0-b3b3-4ac812f1e152 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.797930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c126cb09-7e98-4da0-b3b3-4ac812f1e152 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.799927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c126cb09-7e98-4da0-b3b3-4ac812f1e152 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.815099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1378/2752] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:25.816393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c368c3cf-e3be-4820-a68b-f654d9cda04f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.827876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c368c3cf-e3be-4820-a68b-f654d9cda04f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.834149 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc50a041-b898-4ae6-a93d-3981ef931158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.834958 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc50a041-b898-4ae6-a93d-3981ef931158 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.835410 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc50a041-b898-4ae6-a93d-3981ef931158 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.835636 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc50a041-b898-4ae6-a93d-3981ef931158 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.836377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc50a041-b898-4ae6-a93d-3981ef931158 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.838251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc50a041-b898-4ae6-a93d-3981ef931158 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.850192 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1375/2753] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:25.854783 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c126cb09-7e98-4da0-b3b3-4ac812f1e152 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:25.865664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c126cb09-7e98-4da0-b3b3-4ac812f1e152 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:25.870968 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80954fb3-d229-4c63-a6d2-ca6b8a638f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:25.871745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80954fb3-d229-4c63-a6d2-ca6b8a638f33 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.872118 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80954fb3-d229-4c63-a6d2-ca6b8a638f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.872507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80954fb3-d229-4c63-a6d2-ca6b8a638f33 admin admin] PATH_INFO: `/v3/projects/1e5abfc4040c4203931e1840c5cc211c/users/202253c2df50472f956772a3e01d1180/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.889166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80954fb3-d229-4c63-a6d2-ca6b8a638f33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e5abfc4040c4203931e1840c5cc211c, user_id=202253c2df50472f956772a3e01d1180, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:25.892313 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80954fb3-d229-4c63-a6d2-ca6b8a638f33 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:25.908287 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1379/2754] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:25 2022] PUT /identity/v3/projects/1e5abfc4040c4203931e1840c5cc211c/users/202253c2df50472f956772a3e01d1180/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:25.921035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871cb3de-0ebc-4dcc-8f59-13074a441926 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:25.921696 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871cb3de-0ebc-4dcc-8f59-13074a441926 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:25.922514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871cb3de-0ebc-4dcc-8f59-13074a441926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:25.957752 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-871cb3de-0ebc-4dcc-8f59-13074a441926 None None] MFA Rules not processed for user `202253c2df50472f956772a3e01d1180`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:25.959158 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-871cb3de-0ebc-4dcc-8f59-13074a441926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:26.032722 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1376/2755] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:26.034609 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80954fb3-d229-4c63-a6d2-ca6b8a638f33 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:26.059413 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=4,keystone:SELECT=91,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:26.107715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80954fb3-d229-4c63-a6d2-ca6b8a638f33 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:26.112395 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f462cfe-780c-4fb9-9223-6b3884fda6b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:26.113083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f462cfe-780c-4fb9-9223-6b3884fda6b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:26.113430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f462cfe-780c-4fb9-9223-6b3884fda6b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:26.113755 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f462cfe-780c-4fb9-9223-6b3884fda6b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:26.114489 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f462cfe-780c-4fb9-9223-6b3884fda6b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:26.116462 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f462cfe-780c-4fb9-9223-6b3884fda6b1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:26.150729 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1380/2756] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:26 2022] POST /identity/v3/projects => generated 385 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:26.155138 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871cb3de-0ebc-4dcc-8f59-13074a441926 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:26.167418 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871cb3de-0ebc-4dcc-8f59-13074a441926 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:26.191730 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:26.232481 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78aab61e-9ee4-49a2-bf7d-4171904bd50f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:26.233229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aab61e-9ee4-49a2-bf7d-4171904bd50f admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:26.233577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aab61e-9ee4-49a2-bf7d-4171904bd50f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:26.233916 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aab61e-9ee4-49a2-bf7d-4171904bd50f admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:26.234661 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78aab61e-9ee4-49a2-bf7d-4171904bd50f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:26.236660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78aab61e-9ee4-49a2-bf7d-4171904bd50f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:26.259122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1377/2757] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:26 2022] POST /identity/v3/users => generated 299 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:41:26.263467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f462cfe-780c-4fb9-9223-6b3884fda6b1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:26.274591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f462cfe-780c-4fb9-9223-6b3884fda6b1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:26.279480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc25924-01c8-4211-a49d-09f0f26eeb57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:26.280111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc25924-01c8-4211-a49d-09f0f26eeb57 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:26.280373 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc25924-01c8-4211-a49d-09f0f26eeb57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:26.280637 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc25924-01c8-4211-a49d-09f0f26eeb57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:26.281273 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc25924-01c8-4211-a49d-09f0f26eeb57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:26.282914 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc25924-01c8-4211-a49d-09f0f26eeb57 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:26.293980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1381/2758] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:41:26.298288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78aab61e-9ee4-49a2-bf7d-4171904bd50f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:26.312079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78aab61e-9ee4-49a2-bf7d-4171904bd50f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:26.316780 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8deb26ac-2b45-4b31-b842-f0faee4d2603 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:26.317444 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8deb26ac-2b45-4b31-b842-f0faee4d2603 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:26.317717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8deb26ac-2b45-4b31-b842-f0faee4d2603 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:26.317984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8deb26ac-2b45-4b31-b842-f0faee4d2603 admin admin] PATH_INFO: `/v3/projects/1a6d5c0df02745138f7fb5b15d794e05/users/fb55459ad94b40d790612eaecbcd0010/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:26.335520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8deb26ac-2b45-4b31-b842-f0faee4d2603 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a6d5c0df02745138f7fb5b15d794e05, user_id=fb55459ad94b40d790612eaecbcd0010, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:26.338296 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8deb26ac-2b45-4b31-b842-f0faee4d2603 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:26.352591 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1378/2759] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:26 2022] PUT /identity/v3/projects/1a6d5c0df02745138f7fb5b15d794e05/users/fb55459ad94b40d790612eaecbcd0010/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:41:26.357322 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc25924-01c8-4211-a49d-09f0f26eeb57 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:26.367573 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc25924-01c8-4211-a49d-09f0f26eeb57 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:26.425099 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd329e9-499a-4b8d-9357-3d94de1e5a8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:26.425708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd329e9-499a-4b8d-9357-3d94de1e5a8d admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:26.425986 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd329e9-499a-4b8d-9357-3d94de1e5a8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:26.426282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd329e9-499a-4b8d-9357-3d94de1e5a8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:26.426917 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd329e9-499a-4b8d-9357-3d94de1e5a8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:26.428514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd329e9-499a-4b8d-9357-3d94de1e5a8d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:26.438437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1382/2760] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:26.442907 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8deb26ac-2b45-4b31-b842-f0faee4d2603 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:26.454356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8deb26ac-2b45-4b31-b842-f0faee4d2603 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:26.459918 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa4f453-d0bd-43a5-8c5a-c7bb064f4fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:26.460583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4f453-d0bd-43a5-8c5a-c7bb064f4fbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:26.460861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4f453-d0bd-43a5-8c5a-c7bb064f4fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:26.461130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa4f453-d0bd-43a5-8c5a-c7bb064f4fbe admin admin] PATH_INFO: `/v3/projects/1a6d5c0df02745138f7fb5b15d794e05/users/fb55459ad94b40d790612eaecbcd0010/roles/22819414415942d399db534708fe7103` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:26.463164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa4f453-d0bd-43a5-8c5a-c7bb064f4fbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a6d5c0df02745138f7fb5b15d794e05, user_id=fb55459ad94b40d790612eaecbcd0010, role_id=22819414415942d399db534708fe7103)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:26.465811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa4f453-d0bd-43a5-8c5a-c7bb064f4fbe admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:26.477606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1379/2761] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:26 2022] PUT /identity/v3/projects/1a6d5c0df02745138f7fb5b15d794e05/users/fb55459ad94b40d790612eaecbcd0010/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:26.488168 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3407f61d-eb1f-4d7d-ac32-4f7a7bde0b29 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:26.488276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3407f61d-eb1f-4d7d-ac32-4f7a7bde0b29 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:26.488524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3407f61d-eb1f-4d7d-ac32-4f7a7bde0b29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:26.523109 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-3407f61d-eb1f-4d7d-ac32-4f7a7bde0b29 None None] MFA Rules not processed for user `fb55459ad94b40d790612eaecbcd0010`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:26.524434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3407f61d-eb1f-4d7d-ac32-4f7a7bde0b29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:26.606491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1383/2762] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:26 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:26.619047 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa4f453-d0bd-43a5-8c5a-c7bb064f4fbe admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:26.620795 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9aa4f453-d0bd-43a5-8c5a-c7bb064f4fbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:26.688236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa4f453-d0bd-43a5-8c5a-c7bb064f4fbe admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:26.691927 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f933683-e058-456b-a0ea-4e67356550a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:26.692552 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f933683-e058-456b-a0ea-4e67356550a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:26.692806 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f933683-e058-456b-a0ea-4e67356550a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:26.693072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f933683-e058-456b-a0ea-4e67356550a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:26.769634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f933683-e058-456b-a0ea-4e67356550a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:26.771271 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f933683-e058-456b-a0ea-4e67356550a0 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:26.784602 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1380/2763] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:26 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:27.404863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3407f61d-eb1f-4d7d-ac32-4f7a7bde0b29 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:27.406414 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3407f61d-eb1f-4d7d-ac32-4f7a7bde0b29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:27.425683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:27.474096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3407f61d-eb1f-4d7d-ac32-4f7a7bde0b29 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:27.479250 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:27.479878 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:27.480152 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:27.480418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] PATH_INFO: `/v3/users/f48ce3ac32e741ef98e5025fd23bbaa0` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:27.481551 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f48ce3ac32e741ef98e5025fd23bbaa0)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:27.483495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:27.535225 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f48ce3ac32e741ef98e5025fd23bbaa0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:27.543909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f48ce3ac32e741ef98e5025fd23bbaa0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:27.544562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f48ce3ac32e741ef98e5025fd23bbaa0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:27.545334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f48ce3ac32e741ef98e5025fd23bbaa0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:27.552461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-28f3ce5a-2d9c-43e8-b0f3-043db4bf5881 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f48ce3ac32e741ef98e5025fd23bbaa0'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:27.570202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1384/2764] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:27 2022] DELETE /identity/v3/users/f48ce3ac32e741ef98e5025fd23bbaa0 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:27.752015 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f933683-e058-456b-a0ea-4e67356550a0 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:27.764483 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f933683-e058-456b-a0ea-4e67356550a0 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:27.795020 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:27.824937 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:27.825572 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:27.825853 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:27.826130 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] PATH_INFO: `/v3/projects/76163bf55e0a403aaec08c9c6a68a55f` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:27.827957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] RBAC: Authorizing `identity:delete_project(project_id=76163bf55e0a403aaec08c9c6a68a55f)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:27.831203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:27.885987 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76163bf55e0a403aaec08c9c6a68a55f', 'request_id': 'req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:27.891727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76163bf55e0a403aaec08c9c6a68a55f', 'request_id': 'req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:27.900525 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76163bf55e0a403aaec08c9c6a68a55f', 'request_id': 'req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:27.912731 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] The token cache is being invalidate because project 76163bf55e0a403aaec08c9c6a68a55f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:27.912951 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:27.913646 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-a3de11e7-3e19-4a8a-93eb-f46976bb3c66 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:27.915236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1381/2765] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:27 2022] DELETE /identity/v3/projects/76163bf55e0a403aaec08c9c6a68a55f => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:27.920571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c6fabf-580a-40e7-baec-9440f276edf7 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:27.920884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c6fabf-580a-40e7-baec-9440f276edf7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:27.921176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c6fabf-580a-40e7-baec-9440f276edf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:27.942323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-27c6fabf-580a-40e7-baec-9440f276edf7 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:27.943543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-27c6fabf-580a-40e7-baec-9440f276edf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:28.007525 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1385/2766] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:28.016107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9d338b-8e43-4baf-ac83-a59fe8ea8a05 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.016387 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9d338b-8e43-4baf-ac83-a59fe8ea8a05 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.016655 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9d338b-8e43-4baf-ac83-a59fe8ea8a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.037404 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3f9d338b-8e43-4baf-ac83-a59fe8ea8a05 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:28.038668 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3f9d338b-8e43-4baf-ac83-a59fe8ea8a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:28.047566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1382/2767] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:28.052131 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c6fabf-580a-40e7-baec-9440f276edf7 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.063202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c6fabf-580a-40e7-baec-9440f276edf7 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.070740 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4370245d-0a5c-4e5a-a853-0bcc084388f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.071693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4370245d-0a5c-4e5a-a853-0bcc084388f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.072114 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4370245d-0a5c-4e5a-a853-0bcc084388f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.072673 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4370245d-0a5c-4e5a-a853-0bcc084388f1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.073703 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4370245d-0a5c-4e5a-a853-0bcc084388f1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:41:28.074007 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4370245d-0a5c-4e5a-a853-0bcc084388f1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.075648 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4370245d-0a5c-4e5a-a853-0bcc084388f1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.088798 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1386/2768] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:41:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:41:28.095566 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9d338b-8e43-4baf-ac83-a59fe8ea8a05 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.108193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9d338b-8e43-4baf-ac83-a59fe8ea8a05 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.114543 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4599fb30-0c9c-44ba-9908-aef665398de3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.115156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4599fb30-0c9c-44ba-9908-aef665398de3 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.115437 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4599fb30-0c9c-44ba-9908-aef665398de3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.115706 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4599fb30-0c9c-44ba-9908-aef665398de3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.116363 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4599fb30-0c9c-44ba-9908-aef665398de3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.118352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4599fb30-0c9c-44ba-9908-aef665398de3 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.152228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1383/2769] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:28 2022] POST /identity/v3/projects => generated 379 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:28.153418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4370245d-0a5c-4e5a-a853-0bcc084388f1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.164086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4370245d-0a5c-4e5a-a853-0bcc084388f1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.185054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:28.226177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3086c655-96ab-4175-b32c-636bd83db585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.226805 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3086c655-96ab-4175-b32c-636bd83db585 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.227121 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3086c655-96ab-4175-b32c-636bd83db585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.227381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3086c655-96ab-4175-b32c-636bd83db585 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.227982 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3086c655-96ab-4175-b32c-636bd83db585 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.230428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3086c655-96ab-4175-b32c-636bd83db585 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.253807 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1387/2770] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:28 2022] POST /identity/v3/users => generated 290 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:28.259371 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4599fb30-0c9c-44ba-9908-aef665398de3 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.270379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4599fb30-0c9c-44ba-9908-aef665398de3 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.275067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ea89d7-a03d-4883-b23e-61c727a1ba67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.275774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea89d7-a03d-4883-b23e-61c727a1ba67 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.276057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea89d7-a03d-4883-b23e-61c727a1ba67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.276334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ea89d7-a03d-4883-b23e-61c727a1ba67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.276980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ea89d7-a03d-4883-b23e-61c727a1ba67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.278624 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ea89d7-a03d-4883-b23e-61c727a1ba67 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.289038 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1384/2771] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:28.293265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3086c655-96ab-4175-b32c-636bd83db585 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.304628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3086c655-96ab-4175-b32c-636bd83db585 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.309202 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca68bea-e424-4f25-991b-5bcd43ddc0f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.309811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca68bea-e424-4f25-991b-5bcd43ddc0f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.310082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca68bea-e424-4f25-991b-5bcd43ddc0f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.310394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca68bea-e424-4f25-991b-5bcd43ddc0f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.311031 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca68bea-e424-4f25-991b-5bcd43ddc0f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.312613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca68bea-e424-4f25-991b-5bcd43ddc0f7 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.324194 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1388/2772] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:28.328680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ea89d7-a03d-4883-b23e-61c727a1ba67 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.339672 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ea89d7-a03d-4883-b23e-61c727a1ba67 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.344577 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032b7e4d-1c1c-432e-a045-67030accbfeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.345249 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032b7e4d-1c1c-432e-a045-67030accbfeb admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.345528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032b7e4d-1c1c-432e-a045-67030accbfeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.345804 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032b7e4d-1c1c-432e-a045-67030accbfeb admin admin] PATH_INFO: `/v3/projects/2725a5708b064dea9b8de9c0966c2852/users/c39f1944ed2648fb9e4fc972f9c23b76/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.357767 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca68bea-e424-4f25-991b-5bcd43ddc0f7 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.359686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3ca68bea-e424-4f25-991b-5bcd43ddc0f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:28.361585 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032b7e4d-1c1c-432e-a045-67030accbfeb admin admin] RBAC: Authorizing `identity:create_grant(project_id=2725a5708b064dea9b8de9c0966c2852, user_id=c39f1944ed2648fb9e4fc972f9c23b76, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.364436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032b7e4d-1c1c-432e-a045-67030accbfeb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.380137 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1385/2773] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:28 2022] PUT /identity/v3/projects/2725a5708b064dea9b8de9c0966c2852/users/c39f1944ed2648fb9e4fc972f9c23b76/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:28.392000 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54ebbff-c26b-4378-8c4e-78177c763bdd None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.392280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54ebbff-c26b-4378-8c4e-78177c763bdd None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.392551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54ebbff-c26b-4378-8c4e-78177c763bdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.418491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca68bea-e424-4f25-991b-5bcd43ddc0f7 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.420103 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-b54ebbff-c26b-4378-8c4e-78177c763bdd None None] MFA Rules not processed for user `c39f1944ed2648fb9e4fc972f9c23b76`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:28.421327 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-b54ebbff-c26b-4378-8c4e-78177c763bdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:28.467376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78854ab8-ab18-4af7-b5ab-ac6fe29c1cd8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.468016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78854ab8-ab18-4af7-b5ab-ac6fe29c1cd8 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.468291 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78854ab8-ab18-4af7-b5ab-ac6fe29c1cd8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.468727 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78854ab8-ab18-4af7-b5ab-ac6fe29c1cd8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.469447 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:28.470958 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-78854ab8-ab18-4af7-b5ab-ac6fe29c1cd8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:28.491362 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1386/2774] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:28 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:41:28.496254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54ebbff-c26b-4378-8c4e-78177c763bdd None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.531906 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78854ab8-ab18-4af7-b5ab-ac6fe29c1cd8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.533474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78854ab8-ab18-4af7-b5ab-ac6fe29c1cd8 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.542769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1389/2775] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:28 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:41:28.562326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54ebbff-c26b-4378-8c4e-78177c763bdd None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.567071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7152c2e7-da68-4ab8-8a7d-c419fb73e2ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.567693 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152c2e7-da68-4ab8-8a7d-c419fb73e2ea admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.568128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152c2e7-da68-4ab8-8a7d-c419fb73e2ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.568251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152c2e7-da68-4ab8-8a7d-c419fb73e2ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.568864 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7152c2e7-da68-4ab8-8a7d-c419fb73e2ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.570834 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7152c2e7-da68-4ab8-8a7d-c419fb73e2ea admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.601744 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1387/2776] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:28 2022] POST /identity/v3/projects => generated 381 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:41:28.605960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78854ab8-ab18-4af7-b5ab-ac6fe29c1cd8 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.616966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78854ab8-ab18-4af7-b5ab-ac6fe29c1cd8 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.625909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:28.683721 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e0b66-a127-4a93-9f8a-aa6e6b628e11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.684452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e0b66-a127-4a93-9f8a-aa6e6b628e11 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.684846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e0b66-a127-4a93-9f8a-aa6e6b628e11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.685200 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e0b66-a127-4a93-9f8a-aa6e6b628e11 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.685933 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936e0b66-a127-4a93-9f8a-aa6e6b628e11 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.687992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936e0b66-a127-4a93-9f8a-aa6e6b628e11 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.714281 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1390/2777] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:28 2022] POST /identity/v3/users => generated 297 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:41:28.715694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7152c2e7-da68-4ab8-8a7d-c419fb73e2ea admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.726280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7152c2e7-da68-4ab8-8a7d-c419fb73e2ea admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.731385 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307a913e-ffd8-469b-a316-a0ee3cb51dd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.732013 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307a913e-ffd8-469b-a316-a0ee3cb51dd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.732272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307a913e-ffd8-469b-a316-a0ee3cb51dd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.732546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307a913e-ffd8-469b-a316-a0ee3cb51dd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.733196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307a913e-ffd8-469b-a316-a0ee3cb51dd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.734851 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307a913e-ffd8-469b-a316-a0ee3cb51dd5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.748970 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1388/2778] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:28.750488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e0b66-a127-4a93-9f8a-aa6e6b628e11 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.760794 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e0b66-a127-4a93-9f8a-aa6e6b628e11 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.765834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb28ef8-a032-4819-899a-d66b12f300d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.766537 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb28ef8-a032-4819-899a-d66b12f300d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.766813 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb28ef8-a032-4819-899a-d66b12f300d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.767087 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb28ef8-a032-4819-899a-d66b12f300d1 admin admin] PATH_INFO: `/v3/projects/1e1c4ca46c894727b0d80ea5d736498c/users/34cbf658af544e469283229dec70fac5/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.782541 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb28ef8-a032-4819-899a-d66b12f300d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e1c4ca46c894727b0d80ea5d736498c, user_id=34cbf658af544e469283229dec70fac5, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.785201 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb28ef8-a032-4819-899a-d66b12f300d1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.799786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1391/2779] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:28 2022] PUT /identity/v3/projects/1e1c4ca46c894727b0d80ea5d736498c/users/34cbf658af544e469283229dec70fac5/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:28.803801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307a913e-ffd8-469b-a316-a0ee3cb51dd5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.814814 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307a913e-ffd8-469b-a316-a0ee3cb51dd5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.880389 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c47060-4782-4b92-9455-e6c4505835fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.881121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c47060-4782-4b92-9455-e6c4505835fb admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.881454 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c47060-4782-4b92-9455-e6c4505835fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.881734 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c47060-4782-4b92-9455-e6c4505835fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.882457 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c47060-4782-4b92-9455-e6c4505835fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.884432 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c47060-4782-4b92-9455-e6c4505835fb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.896461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1389/2780] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:28 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:28.901034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb28ef8-a032-4819-899a-d66b12f300d1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:28.911799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb28ef8-a032-4819-899a-d66b12f300d1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:28.917129 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d0389d-5302-4702-b27f-d16580d4c3e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:28.917876 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d0389d-5302-4702-b27f-d16580d4c3e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.918254 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d0389d-5302-4702-b27f-d16580d4c3e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.918613 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d0389d-5302-4702-b27f-d16580d4c3e1 admin admin] PATH_INFO: `/v3/projects/1e1c4ca46c894727b0d80ea5d736498c/users/34cbf658af544e469283229dec70fac5/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.920675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d0389d-5302-4702-b27f-d16580d4c3e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e1c4ca46c894727b0d80ea5d736498c, user_id=34cbf658af544e469283229dec70fac5, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:28.923617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d0389d-5302-4702-b27f-d16580d4c3e1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:28.936294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1392/2781] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:28 2022] PUT /identity/v3/projects/1e1c4ca46c894727b0d80ea5d736498c/users/34cbf658af544e469283229dec70fac5/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:28.948324 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a79ed1b-3579-4358-aad6-70a9211ced8c None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:28.948593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a79ed1b-3579-4358-aad6-70a9211ced8c None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:28.949229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a79ed1b-3579-4358-aad6-70a9211ced8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:28.981179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-2a79ed1b-3579-4358-aad6-70a9211ced8c None None] MFA Rules not processed for user `34cbf658af544e469283229dec70fac5`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:28.982497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2a79ed1b-3579-4358-aad6-70a9211ced8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:29.031204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:29.055793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1390/2782] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:28 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:29.068434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d0389d-5302-4702-b27f-d16580d4c3e1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:29.070404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a0d0389d-5302-4702-b27f-d16580d4c3e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:29.109393 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a79ed1b-3579-4358-aad6-70a9211ced8c None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:29.131737 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d0389d-5302-4702-b27f-d16580d4c3e1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:29.135946 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45ec5a1-2239-4242-b6e7-4966351fe8c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:29.136556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45ec5a1-2239-4242-b6e7-4966351fe8c1 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:29.136975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45ec5a1-2239-4242-b6e7-4966351fe8c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:29.137267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45ec5a1-2239-4242-b6e7-4966351fe8c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:29.146236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45ec5a1-2239-4242-b6e7-4966351fe8c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:29.147691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45ec5a1-2239-4242-b6e7-4966351fe8c1 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:29.157479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1393/2783] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:29 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:29.166209 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a79ed1b-3579-4358-aad6-70a9211ced8c None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:29.170938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6db6f2-3628-4ee1-9464-f6ab0217cd1c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:29.171569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6db6f2-3628-4ee1-9464-f6ab0217cd1c service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:29.171846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6db6f2-3628-4ee1-9464-f6ab0217cd1c service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:29.172111 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6db6f2-3628-4ee1-9464-f6ab0217cd1c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:29.174129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7c6db6f2-3628-4ee1-9464-f6ab0217cd1c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:29.238386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6db6f2-3628-4ee1-9464-f6ab0217cd1c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:29.239862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6db6f2-3628-4ee1-9464-f6ab0217cd1c service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:29.250680 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1391/2784] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:29 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 13 15:41:29.495711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45ec5a1-2239-4242-b6e7-4966351fe8c1 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:29.510426 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45ec5a1-2239-4242-b6e7-4966351fe8c1 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:29.515994 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07ec8e7-6955-4b6a-b11e-7583e45641f0 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34cbf658af544e469283229dec70fac5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1e1c4ca46c894727b0d80ea5d736498c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:29.516687 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07ec8e7-6955-4b6a-b11e-7583e45641f0 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:29.517010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07ec8e7-6955-4b6a-b11e-7583e45641f0 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:29.517310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07ec8e7-6955-4b6a-b11e-7583e45641f0 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:29.518001 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07ec8e7-6955-4b6a-b11e-7583e45641f0 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:41:29.518378 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07ec8e7-6955-4b6a-b11e-7583e45641f0 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:29.520242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07ec8e7-6955-4b6a-b11e-7583e45641f0 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:29.533364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1394/2785] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:41:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:41:29.538329 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6db6f2-3628-4ee1-9464-f6ab0217cd1c service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:29.551809 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6db6f2-3628-4ee1-9464-f6ab0217cd1c service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:29.558066 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c25b69-1890-42ec-8b89-cd235a23891f tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34cbf658af544e469283229dec70fac5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1e1c4ca46c894727b0d80ea5d736498c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:29.558712 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c25b69-1890-42ec-8b89-cd235a23891f tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:29.558992 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c25b69-1890-42ec-8b89-cd235a23891f tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:29.559266 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c25b69-1890-42ec-8b89-cd235a23891f tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:29.559917 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c25b69-1890-42ec-8b89-cd235a23891f tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:29.561848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c25b69-1890-42ec-8b89-cd235a23891f tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:29.594516 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1392/2786] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:29 2022] POST /identity/v3/projects => generated 353 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:29.599824 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07ec8e7-6955-4b6a-b11e-7583e45641f0 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:29.610977 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07ec8e7-6955-4b6a-b11e-7583e45641f0 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:29.663119 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:29.673329 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9541e7-06d0-4b3a-8958-7eb9be3c6537 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34cbf658af544e469283229dec70fac5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1e1c4ca46c894727b0d80ea5d736498c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:29.673945 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9541e7-06d0-4b3a-8958-7eb9be3c6537 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:29.674258 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9541e7-06d0-4b3a-8958-7eb9be3c6537 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:29.674529 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9541e7-06d0-4b3a-8958-7eb9be3c6537 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:29.675177 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9541e7-06d0-4b3a-8958-7eb9be3c6537 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:41:29.675480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9541e7-06d0-4b3a-8958-7eb9be3c6537 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:29.677111 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9541e7-06d0-4b3a-8958-7eb9be3c6537 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:29.688896 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1395/2787] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:41:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:41:29.727682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c25b69-1890-42ec-8b89-cd235a23891f tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:29.739122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c25b69-1890-42ec-8b89-cd235a23891f tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:29.745060 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34cbf658af544e469283229dec70fac5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1e1c4ca46c894727b0d80ea5d736498c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:29.745692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:29.745972 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:29.746291 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] PATH_INFO: `/v3/projects/afca7835646c49b1bd9d2c812e758994` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:29.747479 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: Authorizing `identity:delete_project(project_id=afca7835646c49b1bd9d2c812e758994)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:29.749409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:29.800394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'afca7835646c49b1bd9d2c812e758994', 'request_id': 'req-76afe584-c050-435b-9d04-5921d4c3d848'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:29.805870 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'afca7835646c49b1bd9d2c812e758994', 'request_id': 'req-76afe584-c050-435b-9d04-5921d4c3d848'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:29.814945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'afca7835646c49b1bd9d2c812e758994', 'request_id': 'req-76afe584-c050-435b-9d04-5921d4c3d848'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:29.816107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:29.829422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] The token cache is being invalidate because project afca7835646c49b1bd9d2c812e758994 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:29.829718 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:29.830450 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-76afe584-c050-435b-9d04-5921d4c3d848 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:29.831980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1393/2788] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:29 2022] DELETE /identity/v3/projects/afca7835646c49b1bd9d2c812e758994 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:30.035467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9541e7-06d0-4b3a-8958-7eb9be3c6537 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:30.037411 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-3a9541e7-06d0-4b3a-8958-7eb9be3c6537 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:30.105947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9541e7-06d0-4b3a-8958-7eb9be3c6537 tempest-VolumeQuotaClassesTest-1691310827 tempest-VolumeQuotaClassesTest-1691310827-project-admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:30.171483 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:30.172115 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:30.172383 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:30.172640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] PATH_INFO: `/v3/users/c39f1944ed2648fb9e4fc972f9c23b76` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:30.173875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c39f1944ed2648fb9e4fc972f9c23b76)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:30.175812 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:30.229683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c39f1944ed2648fb9e4fc972f9c23b76'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:30.238147 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c39f1944ed2648fb9e4fc972f9c23b76'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:30.238826 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c39f1944ed2648fb9e4fc972f9c23b76'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:30.239543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c39f1944ed2648fb9e4fc972f9c23b76'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:30.246960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c39f1944ed2648fb9e4fc972f9c23b76'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:30.263811 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1396/2789] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:30 2022] DELETE /identity/v3/users/c39f1944ed2648fb9e4fc972f9c23b76 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:30.268556 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77238f8-5dc6-4c8f-aef0-df709f644f47 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:30.268818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77238f8-5dc6-4c8f-aef0-df709f644f47 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:30.269114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77238f8-5dc6-4c8f-aef0-df709f644f47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:30.288318 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-c77238f8-5dc6-4c8f-aef0-df709f644f47 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:30.289533 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c77238f8-5dc6-4c8f-aef0-df709f644f47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:30.347274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1394/2790] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:30.357593 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:30.359084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:30.375394 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:30.422286 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397bbb65-dd02-4f20-8d93-b4d1aaff5056 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:30.426128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d65768-e22c-47ad-984d-83f0a90a9d38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:30.426748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d65768-e22c-47ad-984d-83f0a90a9d38 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:30.427026 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d65768-e22c-47ad-984d-83f0a90a9d38 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:30.427297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d65768-e22c-47ad-984d-83f0a90a9d38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:30.436838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d65768-e22c-47ad-984d-83f0a90a9d38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:30.438304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d65768-e22c-47ad-984d-83f0a90a9d38 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:30.448066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1397/2791] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:30.775116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77238f8-5dc6-4c8f-aef0-df709f644f47 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:30.776988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c77238f8-5dc6-4c8f-aef0-df709f644f47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:30.793617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77238f8-5dc6-4c8f-aef0-df709f644f47 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:30.799343 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:30.800002 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:30.800268 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:30.800551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] PATH_INFO: `/v3/projects/2725a5708b064dea9b8de9c0966c2852` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:30.801694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2725a5708b064dea9b8de9c0966c2852)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:30.804294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:30.860199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2725a5708b064dea9b8de9c0966c2852', 'request_id': 'req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:30.866524 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2725a5708b064dea9b8de9c0966c2852', 'request_id': 'req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:30.875088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2725a5708b064dea9b8de9c0966c2852', 'request_id': 'req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:30.886942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] The token cache is being invalidate because project 2725a5708b064dea9b8de9c0966c2852 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:30.887233 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:30.887940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:30.889550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1395/2792] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:30 2022] DELETE /identity/v3/projects/2725a5708b064dea9b8de9c0966c2852 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:30.893610 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d65768-e22c-47ad-984d-83f0a90a9d38 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:30.903759 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d65768-e22c-47ad-984d-83f0a90a9d38 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:30.960884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:30.961507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:30.961777 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:30.962030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] PATH_INFO: `/v3/users/34cbf658af544e469283229dec70fac5` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:30.963265 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] RBAC: Authorizing `identity:delete_user(user_id=34cbf658af544e469283229dec70fac5)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:30.965364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:31.015675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34cbf658af544e469283229dec70fac5'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:31.024452 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34cbf658af544e469283229dec70fac5'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:31.025040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34cbf658af544e469283229dec70fac5'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:31.025790 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34cbf658af544e469283229dec70fac5'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:31.032858 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-dac0d1b2-3cae-4da0-8f72-5ac3597a769a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34cbf658af544e469283229dec70fac5'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:31.050644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1398/2793] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:30 2022] DELETE /identity/v3/users/34cbf658af544e469283229dec70fac5 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:31.331460 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:31.333416 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:31.346612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0ad30e-334b-4cc9-9828-7fe9ed14a153 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:31.410356 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:31.410988 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:31.411261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:31.411526 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] PATH_INFO: `/v3/projects/1e1c4ca46c894727b0d80ea5d736498c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:31.412644 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e1c4ca46c894727b0d80ea5d736498c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:31.414587 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:31.435794 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:31.464976 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e1c4ca46c894727b0d80ea5d736498c', 'request_id': 'req-46695d10-26aa-4948-9c62-a4fa6609a774'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:31.470709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e1c4ca46c894727b0d80ea5d736498c', 'request_id': 'req-46695d10-26aa-4948-9c62-a4fa6609a774'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:31.480295 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e1c4ca46c894727b0d80ea5d736498c', 'request_id': 'req-46695d10-26aa-4948-9c62-a4fa6609a774'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:31.491832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] The token cache is being invalidate because project 1e1c4ca46c894727b0d80ea5d736498c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:31.492125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:31.492862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-46695d10-26aa-4948-9c62-a4fa6609a774 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:31.494698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1396/2794] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:31 2022] DELETE /identity/v3/projects/1e1c4ca46c894727b0d80ea5d736498c => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:31.501523 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecc7531-a996-4e20-9746-bd3729ac69c7 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:31.501810 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecc7531-a996-4e20-9746-bd3729ac69c7 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:31.502074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecc7531-a996-4e20-9746-bd3729ac69c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:31.524164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4ecc7531-a996-4e20-9746-bd3729ac69c7 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:31.525393 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4ecc7531-a996-4e20-9746-bd3729ac69c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:31.556300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:31.591354 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1399/2795] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:41:31.599435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a038be67-87d9-4112-8549-f0a3f382b01b None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:31.599720 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a038be67-87d9-4112-8549-f0a3f382b01b None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:31.599991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a038be67-87d9-4112-8549-f0a3f382b01b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:31.621608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-a038be67-87d9-4112-8549-f0a3f382b01b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:31.622861 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-a038be67-87d9-4112-8549-f0a3f382b01b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:31.631991 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1397/2796] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:31.636918 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecc7531-a996-4e20-9746-bd3729ac69c7 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:31.647608 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecc7531-a996-4e20-9746-bd3729ac69c7 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:31.652786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e12f9-bf49-4348-b15b-103f968e284d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:31.653396 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e12f9-bf49-4348-b15b-103f968e284d admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:31.653674 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e12f9-bf49-4348-b15b-103f968e284d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:31.653948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e12f9-bf49-4348-b15b-103f968e284d admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:31.654634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054e12f9-bf49-4348-b15b-103f968e284d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:41:31.654940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054e12f9-bf49-4348-b15b-103f968e284d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:31.656543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054e12f9-bf49-4348-b15b-103f968e284d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:31.668838 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1400/2797] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:41:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:41:31.673532 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a038be67-87d9-4112-8549-f0a3f382b01b None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:31.683494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a038be67-87d9-4112-8549-f0a3f382b01b None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:31.689071 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7629320d-c965-43f4-b52b-9be51872dc06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:31.689670 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629320d-c965-43f4-b52b-9be51872dc06 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:31.689945 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629320d-c965-43f4-b52b-9be51872dc06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:31.690244 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629320d-c965-43f4-b52b-9be51872dc06 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:31.690866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7629320d-c965-43f4-b52b-9be51872dc06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:31.692813 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7629320d-c965-43f4-b52b-9be51872dc06 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:31.724442 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1398/2798] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:31 2022] POST /identity/v3/projects => generated 409 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:31.728866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e12f9-bf49-4348-b15b-103f968e284d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:31.740601 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e12f9-bf49-4348-b15b-103f968e284d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:31.807934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a3244-6407-4abd-ac80-87920dabdf35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:31.808556 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a3244-6407-4abd-ac80-87920dabdf35 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:31.808832 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a3244-6407-4abd-ac80-87920dabdf35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:31.809124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a3244-6407-4abd-ac80-87920dabdf35 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:31.809789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879a3244-6407-4abd-ac80-87920dabdf35 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:31.811755 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879a3244-6407-4abd-ac80-87920dabdf35 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:31.838864 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1401/2799] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:31 2022] POST /identity/v3/users => generated 305 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:41:31.840569 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7629320d-c965-43f4-b52b-9be51872dc06 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:31.856593 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7629320d-c965-43f4-b52b-9be51872dc06 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:31.861467 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f5cf87-7264-4a95-a229-780cf118bd28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:31.862088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f5cf87-7264-4a95-a229-780cf118bd28 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:31.862426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f5cf87-7264-4a95-a229-780cf118bd28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:31.862694 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f5cf87-7264-4a95-a229-780cf118bd28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:31.863331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f5cf87-7264-4a95-a229-780cf118bd28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:31.864938 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f5cf87-7264-4a95-a229-780cf118bd28 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:31.875239 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1399/2800] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:31 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:31.879348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a3244-6407-4abd-ac80-87920dabdf35 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:31.889536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a3244-6407-4abd-ac80-87920dabdf35 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:31.894045 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ea4567-6f8a-4a2f-94bc-aaafd70a3eac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:31.894677 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ea4567-6f8a-4a2f-94bc-aaafd70a3eac admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:31.894950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ea4567-6f8a-4a2f-94bc-aaafd70a3eac admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:31.895197 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ea4567-6f8a-4a2f-94bc-aaafd70a3eac admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:31.895837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ea4567-6f8a-4a2f-94bc-aaafd70a3eac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:31.897437 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ea4567-6f8a-4a2f-94bc-aaafd70a3eac admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:31.908559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1402/2801] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:31.912999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f5cf87-7264-4a95-a229-780cf118bd28 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:31.924275 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f5cf87-7264-4a95-a229-780cf118bd28 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:31.929061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bd1057-4a5f-4bc6-8211-f5382228f824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:31.929714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bd1057-4a5f-4bc6-8211-f5382228f824 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:31.929985 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bd1057-4a5f-4bc6-8211-f5382228f824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:31.931592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bd1057-4a5f-4bc6-8211-f5382228f824 admin admin] PATH_INFO: `/v3/projects/4c5c1cf00fe74cccb83f969e16eb0e5d/users/c46ec43e93f14ca7bf3cb87b006d832e/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:31.946164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bd1057-4a5f-4bc6-8211-f5382228f824 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c5c1cf00fe74cccb83f969e16eb0e5d, user_id=c46ec43e93f14ca7bf3cb87b006d832e, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:31.949294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bd1057-4a5f-4bc6-8211-f5382228f824 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:31.965642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1400/2802] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:31 2022] PUT /identity/v3/projects/4c5c1cf00fe74cccb83f969e16eb0e5d/users/c46ec43e93f14ca7bf3cb87b006d832e/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:31.977591 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9af58e4-ebe3-495d-92bd-f011d05523ee None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:31.977867 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9af58e4-ebe3-495d-92bd-f011d05523ee None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:31.978135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9af58e4-ebe3-495d-92bd-f011d05523ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:32.008004 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a9af58e4-ebe3-495d-92bd-f011d05523ee None None] MFA Rules not processed for user `c46ec43e93f14ca7bf3cb87b006d832e`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:32.009236 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a9af58e4-ebe3-495d-92bd-f011d05523ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:32.081978 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1403/2803] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:31 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:32.086859 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bd1057-4a5f-4bc6-8211-f5382228f824 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:32.148280 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bd1057-4a5f-4bc6-8211-f5382228f824 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:32.153511 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b845e4-fcdf-436b-a439-cc06473c0862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:32.154129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b845e4-fcdf-436b-a439-cc06473c0862 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:32.154421 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b845e4-fcdf-436b-a439-cc06473c0862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:32.154690 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b845e4-fcdf-436b-a439-cc06473c0862 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:32.155320 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b845e4-fcdf-436b-a439-cc06473c0862 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:32.157253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b845e4-fcdf-436b-a439-cc06473c0862 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:32.187650 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1401/2804] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:32 2022] POST /identity/v3/projects => generated 407 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:32.192708 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9af58e4-ebe3-495d-92bd-f011d05523ee None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:32.208728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9af58e4-ebe3-495d-92bd-f011d05523ee None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:32.241931 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:32.275198 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e430eff5-e1a8-483d-aa95-d879c8f759e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:32.275995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e430eff5-e1a8-483d-aa95-d879c8f759e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:32.276408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e430eff5-e1a8-483d-aa95-d879c8f759e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:32.276659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e430eff5-e1a8-483d-aa95-d879c8f759e8 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:32.277297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e430eff5-e1a8-483d-aa95-d879c8f759e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:32.279310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e430eff5-e1a8-483d-aa95-d879c8f759e8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:32.306408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1404/2805] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:32 2022] POST /identity/v3/users => generated 310 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:32.307970 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b845e4-fcdf-436b-a439-cc06473c0862 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:32.318669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b845e4-fcdf-436b-a439-cc06473c0862 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:32.323828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8151206-4513-4ca4-9bd5-594325341cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:32.324422 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8151206-4513-4ca4-9bd5-594325341cb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:32.324714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8151206-4513-4ca4-9bd5-594325341cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:32.324985 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8151206-4513-4ca4-9bd5-594325341cb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:32.325614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8151206-4513-4ca4-9bd5-594325341cb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:32.327273 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8151206-4513-4ca4-9bd5-594325341cb1 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:32.339288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1402/2806] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:32.343901 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e430eff5-e1a8-483d-aa95-d879c8f759e8 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:32.354318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e430eff5-e1a8-483d-aa95-d879c8f759e8 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:32.359255 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6df6e8-c399-4c03-8868-529a3e59f60a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:32.359907 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6df6e8-c399-4c03-8868-529a3e59f60a admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:32.360216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6df6e8-c399-4c03-8868-529a3e59f60a admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:32.360484 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6df6e8-c399-4c03-8868-529a3e59f60a admin admin] PATH_INFO: `/v3/projects/f84e3d0ce448461dab8fa86512be157b/users/0de5621b600740a6806ef40088b47305/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:32.375971 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6df6e8-c399-4c03-8868-529a3e59f60a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f84e3d0ce448461dab8fa86512be157b, user_id=0de5621b600740a6806ef40088b47305, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:32.378651 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6df6e8-c399-4c03-8868-529a3e59f60a admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:32.393595 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1405/2807] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:32 2022] PUT /identity/v3/projects/f84e3d0ce448461dab8fa86512be157b/users/0de5621b600740a6806ef40088b47305/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:32.398523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8151206-4513-4ca4-9bd5-594325341cb1 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:32.413688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8151206-4513-4ca4-9bd5-594325341cb1 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:32.426749 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:32.472443 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc4ae3b-ba6a-4fb9-aac5-b897608f43d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:32.473051 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc4ae3b-ba6a-4fb9-aac5-b897608f43d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:32.473325 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc4ae3b-ba6a-4fb9-aac5-b897608f43d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:32.473596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc4ae3b-ba6a-4fb9-aac5-b897608f43d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:32.474260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc4ae3b-ba6a-4fb9-aac5-b897608f43d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:32.475876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc4ae3b-ba6a-4fb9-aac5-b897608f43d8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:32.486317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1403/2808] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:32 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:32.490668 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6df6e8-c399-4c03-8868-529a3e59f60a admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:32.501228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6df6e8-c399-4c03-8868-529a3e59f60a admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:32.506453 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012fc9f5-0f36-457c-9a59-6c98680605d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:32.507108 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012fc9f5-0f36-457c-9a59-6c98680605d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:32.507378 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012fc9f5-0f36-457c-9a59-6c98680605d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:32.507640 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012fc9f5-0f36-457c-9a59-6c98680605d1 admin admin] PATH_INFO: `/v3/projects/f84e3d0ce448461dab8fa86512be157b/users/0de5621b600740a6806ef40088b47305/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:32.509815 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012fc9f5-0f36-457c-9a59-6c98680605d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f84e3d0ce448461dab8fa86512be157b, user_id=0de5621b600740a6806ef40088b47305, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:32.512750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012fc9f5-0f36-457c-9a59-6c98680605d1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:32.523920 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1406/2809] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:32 2022] PUT /identity/v3/projects/f84e3d0ce448461dab8fa86512be157b/users/0de5621b600740a6806ef40088b47305/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:32.534841 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d0ff0d-1255-4372-85bc-b6cadae05de4 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:32.535272 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d0ff0d-1255-4372-85bc-b6cadae05de4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:32.535446 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d0ff0d-1255-4372-85bc-b6cadae05de4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:32.564021 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-65d0ff0d-1255-4372-85bc-b6cadae05de4 None None] MFA Rules not processed for user `0de5621b600740a6806ef40088b47305`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:32.565235 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-65d0ff0d-1255-4372-85bc-b6cadae05de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:32.635906 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1404/2810] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:32 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:32.646932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012fc9f5-0f36-457c-9a59-6c98680605d1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:32.649060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-012fc9f5-0f36-457c-9a59-6c98680605d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:32.712749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012fc9f5-0f36-457c-9a59-6c98680605d1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:32.717307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72dbffc-30ea-401c-9f07-7b1a0807c941 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:32.717909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72dbffc-30ea-401c-9f07-7b1a0807c941 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:32.718293 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72dbffc-30ea-401c-9f07-7b1a0807c941 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:32.718554 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72dbffc-30ea-401c-9f07-7b1a0807c941 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:32.729429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72dbffc-30ea-401c-9f07-7b1a0807c941 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:32.730871 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72dbffc-30ea-401c-9f07-7b1a0807c941 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:32.740625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1407/2811] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:32 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:32.910862 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d0ff0d-1255-4372-85bc-b6cadae05de4 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:32.922311 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d0ff0d-1255-4372-85bc-b6cadae05de4 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:32.927253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3e77b7-af0a-487e-b77c-1f0ac95775ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:32.927852 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3e77b7-af0a-487e-b77c-1f0ac95775ec service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:32.928125 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3e77b7-af0a-487e-b77c-1f0ac95775ec service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:32.928396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3e77b7-af0a-487e-b77c-1f0ac95775ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:32.994632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3e77b7-af0a-487e-b77c-1f0ac95775ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:32.996268 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3e77b7-af0a-487e-b77c-1f0ac95775ec service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:33.005505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1405/2812] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:32 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:33.186684 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72dbffc-30ea-401c-9f07-7b1a0807c941 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:33.188238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a72dbffc-30ea-401c-9f07-7b1a0807c941 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:33.258985 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72dbffc-30ea-401c-9f07-7b1a0807c941 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:33.263935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:33.264579 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:33.264855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:33.265126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] PATH_INFO: `/v3/users/b8b58ad122794b0db0443733d342bad8` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:33.266276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8b58ad122794b0db0443733d342bad8)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:33.268219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:33.302363 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:33.332557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8b58ad122794b0db0443733d342bad8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:33.342603 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8b58ad122794b0db0443733d342bad8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:33.343228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8b58ad122794b0db0443733d342bad8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:33.343829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8b58ad122794b0db0443733d342bad8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:33.350965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8b58ad122794b0db0443733d342bad8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:33.369223 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1408/2813] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:33 2022] DELETE /identity/v3/users/b8b58ad122794b0db0443733d342bad8 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:33.374185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a88b4d8-36f0-4f99-8a69-c2bc1c0b08e9 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:33.374497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a88b4d8-36f0-4f99-8a69-c2bc1c0b08e9 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:33.374774 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a88b4d8-36f0-4f99-8a69-c2bc1c0b08e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:33.393921 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-7a88b4d8-36f0-4f99-8a69-c2bc1c0b08e9 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:33.395171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7a88b4d8-36f0-4f99-8a69-c2bc1c0b08e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:33.426926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:33.455506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1406/2814] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:33.466320 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:33.467750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:33.530884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db8da71-cc92-4310-8911-40f9d34ed3e8 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:33.535317 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b00d26c-2567-4a6d-9d7f-ac7b96339e05 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:33.535936 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b00d26c-2567-4a6d-9d7f-ac7b96339e05 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:33.536213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b00d26c-2567-4a6d-9d7f-ac7b96339e05 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:33.536504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b00d26c-2567-4a6d-9d7f-ac7b96339e05 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:33.548659 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b00d26c-2567-4a6d-9d7f-ac7b96339e05 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:33.550138 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b00d26c-2567-4a6d-9d7f-ac7b96339e05 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:33.561333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1409/2815] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:33.867948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a88b4d8-36f0-4f99-8a69-c2bc1c0b08e9 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:33.869381 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-7a88b4d8-36f0-4f99-8a69-c2bc1c0b08e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:33.883213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a88b4d8-36f0-4f99-8a69-c2bc1c0b08e9 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:33.888793 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:33.889538 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:33.889826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:33.890139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] PATH_INFO: `/v3/projects/2a3041f6bdbf427aa10b3923355f6985` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:33.892082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a3041f6bdbf427aa10b3923355f6985)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:33.894519 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:33.944234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a3041f6bdbf427aa10b3923355f6985', 'request_id': 'req-1972731a-14fc-4a2a-838f-9f59f4705af5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:33.949947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a3041f6bdbf427aa10b3923355f6985', 'request_id': 'req-1972731a-14fc-4a2a-838f-9f59f4705af5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:33.959512 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a3041f6bdbf427aa10b3923355f6985', 'request_id': 'req-1972731a-14fc-4a2a-838f-9f59f4705af5'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:33.971708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] The token cache is being invalidate because project 2a3041f6bdbf427aa10b3923355f6985 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:33.971996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:33.972639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:33.974173 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1407/2816] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:33 2022] DELETE /identity/v3/projects/2a3041f6bdbf427aa10b3923355f6985 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:33.978189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b00d26c-2567-4a6d-9d7f-ac7b96339e05 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:33.988534 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b00d26c-2567-4a6d-9d7f-ac7b96339e05 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:34.059632 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:34.060267 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:34.060540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:34.060809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] PATH_INFO: `/v3/users/202253c2df50472f956772a3e01d1180` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:34.062040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=202253c2df50472f956772a3e01d1180)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:34.063964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:34.114784 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '202253c2df50472f956772a3e01d1180'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.117261 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:34.124196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '202253c2df50472f956772a3e01d1180'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.124806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '202253c2df50472f956772a3e01d1180'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.125424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '202253c2df50472f956772a3e01d1180'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.137884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '202253c2df50472f956772a3e01d1180'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.156089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1410/2817] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:33 2022] DELETE /identity/v3/users/202253c2df50472f956772a3e01d1180 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:34.435212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:34.436745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:34.456082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1972731a-14fc-4a2a-838f-9f59f4705af5 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:34.523989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:34.524546 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:34.524822 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:34.525093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] PATH_INFO: `/v3/projects/1e5abfc4040c4203931e1840c5cc211c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:34.526260 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e5abfc4040c4203931e1840c5cc211c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:34.528707 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:34.589380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e5abfc4040c4203931e1840c5cc211c', 'request_id': 'req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.595024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e5abfc4040c4203931e1840c5cc211c', 'request_id': 'req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.600377 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:34.604928 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e5abfc4040c4203931e1840c5cc211c', 'request_id': 'req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.618129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] The token cache is being invalidate because project 1e5abfc4040c4203931e1840c5cc211c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:34.618413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.619107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.620741 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1408/2818] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:34 2022] DELETE /identity/v3/projects/1e5abfc4040c4203931e1840c5cc211c => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:34.624932 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:34.626432 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:34.639381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96971bc-701c-454b-b43b-c66bac158d91 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:34.697046 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:34.697688 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:34.697969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:34.698276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] PATH_INFO: `/v3/users/fb55459ad94b40d790612eaecbcd0010` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:34.699418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb55459ad94b40d790612eaecbcd0010)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:34.701389 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:34.753572 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb55459ad94b40d790612eaecbcd0010'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.762629 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb55459ad94b40d790612eaecbcd0010'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.763242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb55459ad94b40d790612eaecbcd0010'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.763875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb55459ad94b40d790612eaecbcd0010'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.770721 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f28822c6-40ef-4f63-85f7-0def974f7119 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb55459ad94b40d790612eaecbcd0010'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:34.788746 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1411/2819] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:34 2022] DELETE /identity/v3/users/fb55459ad94b40d790612eaecbcd0010 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:35.071185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:35.072713 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:35.088943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8585275-ab0a-4d48-bb11-91b04d0ac2ae admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:35.150779 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:35.151423 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.151703 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.151974 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] PATH_INFO: `/v3/projects/1a6d5c0df02745138f7fb5b15d794e05` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.153247 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a6d5c0df02745138f7fb5b15d794e05)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:35.155228 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:35.221957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a6d5c0df02745138f7fb5b15d794e05', 'request_id': 'req-8a65435f-baf6-46a8-b85f-678d99d76489'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:35.227936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a6d5c0df02745138f7fb5b15d794e05', 'request_id': 'req-8a65435f-baf6-46a8-b85f-678d99d76489'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:35.237733 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a6d5c0df02745138f7fb5b15d794e05', 'request_id': 'req-8a65435f-baf6-46a8-b85f-678d99d76489'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:35.251697 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] The token cache is being invalidate because project 1a6d5c0df02745138f7fb5b15d794e05 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:35.252075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:35.253116 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-8a65435f-baf6-46a8-b85f-678d99d76489 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:35.255594 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1409/2820] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:35 2022] DELETE /identity/v3/projects/1a6d5c0df02745138f7fb5b15d794e05 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:35.263691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccb67b7-c615-4152-b40c-0eb3c15679ed None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.264002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccb67b7-c615-4152-b40c-0eb3c15679ed None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.264360 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccb67b7-c615-4152-b40c-0eb3c15679ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.286145 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-5ccb67b7-c615-4152-b40c-0eb3c15679ed None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:35.287428 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5ccb67b7-c615-4152-b40c-0eb3c15679ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:35.339487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:35.361172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1412/2821] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:35.369012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4391b6c7-035b-4f0b-86a2-b755b08bda67 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.369301 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4391b6c7-035b-4f0b-86a2-b755b08bda67 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.369571 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4391b6c7-035b-4f0b-86a2-b755b08bda67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.390981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-4391b6c7-035b-4f0b-86a2-b755b08bda67 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:35.392204 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4391b6c7-035b-4f0b-86a2-b755b08bda67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:35.401333 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1410/2822] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:35.405929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccb67b7-c615-4152-b40c-0eb3c15679ed None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:35.416145 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccb67b7-c615-4152-b40c-0eb3c15679ed None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:35.421853 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d54ee-9bf2-46a6-b0a3-c7f20593c257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:35.422492 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d54ee-9bf2-46a6-b0a3-c7f20593c257 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.422765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d54ee-9bf2-46a6-b0a3-c7f20593c257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.423030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323d54ee-9bf2-46a6-b0a3-c7f20593c257 admin admin] PATH_INFO: `/v3/domains` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.423697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323d54ee-9bf2-46a6-b0a3-c7f20593c257 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81455) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:41:35.423996 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323d54ee-9bf2-46a6-b0a3-c7f20593c257 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:35.425607 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323d54ee-9bf2-46a6-b0a3-c7f20593c257 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:35.437364 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1413/2823] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:41:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:41:35.441832 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4391b6c7-035b-4f0b-86a2-b755b08bda67 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:35.451304 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4391b6c7-035b-4f0b-86a2-b755b08bda67 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:35.456893 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ae9556-ca48-456a-913f-6eaccc50af1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:35.457496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ae9556-ca48-456a-913f-6eaccc50af1f admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.457772 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ae9556-ca48-456a-913f-6eaccc50af1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.458040 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ae9556-ca48-456a-913f-6eaccc50af1f admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.459076 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ae9556-ca48-456a-913f-6eaccc50af1f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:35.461003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ae9556-ca48-456a-913f-6eaccc50af1f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:35.491986 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1411/2824] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:35 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:41:35.496480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d54ee-9bf2-46a6-b0a3-c7f20593c257 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:35.506392 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323d54ee-9bf2-46a6-b0a3-c7f20593c257 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:35.573941 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ced5d4-0dae-4d8c-a84b-87e1e39a82ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:35.574728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ced5d4-0dae-4d8c-a84b-87e1e39a82ed admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.575012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ced5d4-0dae-4d8c-a84b-87e1e39a82ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.575288 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ced5d4-0dae-4d8c-a84b-87e1e39a82ed admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.575944 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ced5d4-0dae-4d8c-a84b-87e1e39a82ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:35.577861 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ced5d4-0dae-4d8c-a84b-87e1e39a82ed admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:35.602091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1414/2825] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:35 2022] POST /identity/v3/users => generated 292 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:35.606726 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ae9556-ca48-456a-913f-6eaccc50af1f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:35.623012 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ae9556-ca48-456a-913f-6eaccc50af1f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:35.628490 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c134b40-05b8-4ba9-b375-a7d9250936b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:35.629117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c134b40-05b8-4ba9-b375-a7d9250936b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.629392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c134b40-05b8-4ba9-b375-a7d9250936b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.629660 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c134b40-05b8-4ba9-b375-a7d9250936b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.630409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c134b40-05b8-4ba9-b375-a7d9250936b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:35.632062 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c134b40-05b8-4ba9-b375-a7d9250936b2 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:35.642887 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1412/2826] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:35 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:35.647642 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ced5d4-0dae-4d8c-a84b-87e1e39a82ed admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:35.658361 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ced5d4-0dae-4d8c-a84b-87e1e39a82ed admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:35.663241 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7f9a26-c6f9-4c4e-8b52-ae72f49d6596 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:35.663835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7f9a26-c6f9-4c4e-8b52-ae72f49d6596 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.664107 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7f9a26-c6f9-4c4e-8b52-ae72f49d6596 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.664372 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7f9a26-c6f9-4c4e-8b52-ae72f49d6596 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.665000 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7f9a26-c6f9-4c4e-8b52-ae72f49d6596 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:35.666623 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7f9a26-c6f9-4c4e-8b52-ae72f49d6596 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:35.677427 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1415/2827] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:35.681656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c134b40-05b8-4ba9-b375-a7d9250936b2 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:35.691970 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c134b40-05b8-4ba9-b375-a7d9250936b2 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:35.697621 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c5b84-550b-4ec3-9011-b1d0b0c99230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:35.698300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c5b84-550b-4ec3-9011-b1d0b0c99230 admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.698574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c5b84-550b-4ec3-9011-b1d0b0c99230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.698842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c5b84-550b-4ec3-9011-b1d0b0c99230 admin admin] PATH_INFO: `/v3/projects/44c95c5ba3cf49e78c1c8a3f98a5752e/users/1257686c014a418098d97831461030be/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.714142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738c5b84-550b-4ec3-9011-b1d0b0c99230 admin admin] RBAC: Authorizing `identity:create_grant(project_id=44c95c5ba3cf49e78c1c8a3f98a5752e, user_id=1257686c014a418098d97831461030be, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:35.716824 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738c5b84-550b-4ec3-9011-b1d0b0c99230 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:35.731207 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1413/2828] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:35 2022] PUT /identity/v3/projects/44c95c5ba3cf49e78c1c8a3f98a5752e/users/1257686c014a418098d97831461030be/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:35.742692 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a64b1bf-66fd-487b-8f0a-50fa6336721d None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.742969 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a64b1bf-66fd-487b-8f0a-50fa6336721d None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.743237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a64b1bf-66fd-487b-8f0a-50fa6336721d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.771622 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4a64b1bf-66fd-487b-8f0a-50fa6336721d None None] MFA Rules not processed for user `1257686c014a418098d97831461030be`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:35.772849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4a64b1bf-66fd-487b-8f0a-50fa6336721d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:35.839071 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1416/2829] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:35.844078 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c5b84-550b-4ec3-9011-b1d0b0c99230 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:35.863977 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:35.913980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c5b84-550b-4ec3-9011-b1d0b0c99230 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:35.919211 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda4f537-3ec4-487f-bb0c-82638225cfd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:35.919820 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda4f537-3ec4-487f-bb0c-82638225cfd2 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:35.920099 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda4f537-3ec4-487f-bb0c-82638225cfd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:35.920379 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda4f537-3ec4-487f-bb0c-82638225cfd2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:35.921049 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda4f537-3ec4-487f-bb0c-82638225cfd2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:35.923083 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda4f537-3ec4-487f-bb0c-82638225cfd2 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:35.951995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1414/2830] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:35 2022] POST /identity/v3/projects => generated 383 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:35.956369 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a64b1bf-66fd-487b-8f0a-50fa6336721d None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:35.968998 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a64b1bf-66fd-487b-8f0a-50fa6336721d None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.010348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:36.028693 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c8b965-e5ee-4205-8532-5c68cd409f4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.029345 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c8b965-e5ee-4205-8532-5c68cd409f4d admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.029624 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c8b965-e5ee-4205-8532-5c68cd409f4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.029898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c8b965-e5ee-4205-8532-5c68cd409f4d admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.030580 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c8b965-e5ee-4205-8532-5c68cd409f4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:36.032628 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c8b965-e5ee-4205-8532-5c68cd409f4d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:36.058997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1417/2831] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:35 2022] POST /identity/v3/users => generated 292 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:41:36.063979 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda4f537-3ec4-487f-bb0c-82638225cfd2 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:36.078502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda4f537-3ec4-487f-bb0c-82638225cfd2 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.084213 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa94b148-191a-40e1-b0ff-18aa04feed41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.084818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa94b148-191a-40e1-b0ff-18aa04feed41 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.085098 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa94b148-191a-40e1-b0ff-18aa04feed41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.085367 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa94b148-191a-40e1-b0ff-18aa04feed41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.086011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa94b148-191a-40e1-b0ff-18aa04feed41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:36.087746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa94b148-191a-40e1-b0ff-18aa04feed41 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:36.098530 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1415/2832] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:36 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:36.102574 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c8b965-e5ee-4205-8532-5c68cd409f4d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:36.113715 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c8b965-e5ee-4205-8532-5c68cd409f4d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.119965 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d19e0e-1641-4d50-b0a7-42b60940a32c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.120577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d19e0e-1641-4d50-b0a7-42b60940a32c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.120863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d19e0e-1641-4d50-b0a7-42b60940a32c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.121280 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d19e0e-1641-4d50-b0a7-42b60940a32c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.121991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d19e0e-1641-4d50-b0a7-42b60940a32c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:36.123630 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d19e0e-1641-4d50-b0a7-42b60940a32c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:36.134840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1418/2833] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:41:36.139170 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa94b148-191a-40e1-b0ff-18aa04feed41 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:36.149656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa94b148-191a-40e1-b0ff-18aa04feed41 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.155374 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f4bf85-4d5f-4fb9-a77d-06c5e2a43c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.156028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f4bf85-4d5f-4fb9-a77d-06c5e2a43c0e admin admin] REQUEST_METHOD: `PUT` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.156309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f4bf85-4d5f-4fb9-a77d-06c5e2a43c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.156584 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f4bf85-4d5f-4fb9-a77d-06c5e2a43c0e admin admin] PATH_INFO: `/v3/projects/265aa1982e534368a6f631fce4915560/users/8c7a720614434c72837d82cba2ef6cbc/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.172292 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f4bf85-4d5f-4fb9-a77d-06c5e2a43c0e admin admin] RBAC: Authorizing `identity:create_grant(project_id=265aa1982e534368a6f631fce4915560, user_id=8c7a720614434c72837d82cba2ef6cbc, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:36.175079 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f4bf85-4d5f-4fb9-a77d-06c5e2a43c0e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:36.190797 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1416/2834] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:36 2022] PUT /identity/v3/projects/265aa1982e534368a6f631fce4915560/users/8c7a720614434c72837d82cba2ef6cbc/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:36.205145 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cf6fe9-7e1c-4054-a480-717c793d9e61 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.205430 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cf6fe9-7e1c-4054-a480-717c793d9e61 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.205698 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cf6fe9-7e1c-4054-a480-717c793d9e61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.238056 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-37cf6fe9-7e1c-4054-a480-717c793d9e61 None None] MFA Rules not processed for user `8c7a720614434c72837d82cba2ef6cbc`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:36.239306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-37cf6fe9-7e1c-4054-a480-717c793d9e61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:36.305643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1419/2835] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:36 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:36.310984 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f4bf85-4d5f-4fb9-a77d-06c5e2a43c0e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:36.373459 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f4bf85-4d5f-4fb9-a77d-06c5e2a43c0e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.379166 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2032476-7fc4-490b-a147-6f6c9619cfa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.379763 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2032476-7fc4-490b-a147-6f6c9619cfa8 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.380064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2032476-7fc4-490b-a147-6f6c9619cfa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.380328 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2032476-7fc4-490b-a147-6f6c9619cfa8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.380955 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2032476-7fc4-490b-a147-6f6c9619cfa8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:36.382926 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2032476-7fc4-490b-a147-6f6c9619cfa8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:36.412107 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1417/2836] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:36 2022] POST /identity/v3/projects => generated 385 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:36.416335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cf6fe9-7e1c-4054-a480-717c793d9e61 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:36.427377 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cf6fe9-7e1c-4054-a480-717c793d9e61 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.488333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1d3258-a8ad-41de-9e43-3a7bb9f3fa67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.488948 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d3258-a8ad-41de-9e43-3a7bb9f3fa67 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.489224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d3258-a8ad-41de-9e43-3a7bb9f3fa67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.489492 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1d3258-a8ad-41de-9e43-3a7bb9f3fa67 admin admin] PATH_INFO: `/v3/users` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.490130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1d3258-a8ad-41de-9e43-3a7bb9f3fa67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:36.492058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1d3258-a8ad-41de-9e43-3a7bb9f3fa67 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:36.514893 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1420/2837] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:36 2022] POST /identity/v3/users => generated 299 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:36.519082 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2032476-7fc4-490b-a147-6f6c9619cfa8 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:36.530146 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2032476-7fc4-490b-a147-6f6c9619cfa8 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.535868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a0bf9c-4f6e-41dc-95e7-c3451e24aa43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.536567 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a0bf9c-4f6e-41dc-95e7-c3451e24aa43 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.536980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a0bf9c-4f6e-41dc-95e7-c3451e24aa43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.537183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a0bf9c-4f6e-41dc-95e7-c3451e24aa43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.537867 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a0bf9c-4f6e-41dc-95e7-c3451e24aa43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:36.539535 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a0bf9c-4f6e-41dc-95e7-c3451e24aa43 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:36.550358 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1418/2838] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:36.554643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1d3258-a8ad-41de-9e43-3a7bb9f3fa67 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:36.564753 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1d3258-a8ad-41de-9e43-3a7bb9f3fa67 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.570398 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e6f6fa-a8b4-40eb-8f4b-6f9796f38480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.571054 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6f6fa-a8b4-40eb-8f4b-6f9796f38480 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.571329 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6f6fa-a8b4-40eb-8f4b-6f9796f38480 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.571594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e6f6fa-a8b4-40eb-8f4b-6f9796f38480 admin admin] PATH_INFO: `/v3/projects/8b8310501ef74b3fa72e1d86e5ae97c6/users/fc3f382490974ef1985df8878dd6a054/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.587511 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e6f6fa-a8b4-40eb-8f4b-6f9796f38480 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b8310501ef74b3fa72e1d86e5ae97c6, user_id=fc3f382490974ef1985df8878dd6a054, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:36.590292 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e6f6fa-a8b4-40eb-8f4b-6f9796f38480 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:36.606180 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1421/2839] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:36 2022] PUT /identity/v3/projects/8b8310501ef74b3fa72e1d86e5ae97c6/users/fc3f382490974ef1985df8878dd6a054/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:41:36.610236 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a0bf9c-4f6e-41dc-95e7-c3451e24aa43 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:36.620997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a0bf9c-4f6e-41dc-95e7-c3451e24aa43 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.669654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:36.687819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db471a8f-27e7-440d-a963-2810abeb8483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.688485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db471a8f-27e7-440d-a963-2810abeb8483 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.688788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db471a8f-27e7-440d-a963-2810abeb8483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.689114 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db471a8f-27e7-440d-a963-2810abeb8483 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.689766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db471a8f-27e7-440d-a963-2810abeb8483 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:36.691451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db471a8f-27e7-440d-a963-2810abeb8483 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:36.702074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1419/2840] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:36 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:36.706627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e6f6fa-a8b4-40eb-8f4b-6f9796f38480 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:36.719222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e6f6fa-a8b4-40eb-8f4b-6f9796f38480 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.724927 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2064a1-520d-4f85-8c31-e26d0a6ad214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.725594 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2064a1-520d-4f85-8c31-e26d0a6ad214 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.725930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2064a1-520d-4f85-8c31-e26d0a6ad214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.726308 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2064a1-520d-4f85-8c31-e26d0a6ad214 admin admin] PATH_INFO: `/v3/projects/8b8310501ef74b3fa72e1d86e5ae97c6/users/fc3f382490974ef1985df8878dd6a054/roles/22819414415942d399db534708fe7103` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.728381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2064a1-520d-4f85-8c31-e26d0a6ad214 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b8310501ef74b3fa72e1d86e5ae97c6, user_id=fc3f382490974ef1985df8878dd6a054, role_id=22819414415942d399db534708fe7103)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:36.731297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2064a1-520d-4f85-8c31-e26d0a6ad214 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:36.742272 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1422/2841] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:36 2022] PUT /identity/v3/projects/8b8310501ef74b3fa72e1d86e5ae97c6/users/fc3f382490974ef1985df8878dd6a054/roles/22819414415942d399db534708fe7103 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:36.754018 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97a22f-0128-42fb-a735-c16c6f6c5b12 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.754468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97a22f-0128-42fb-a735-c16c6f6c5b12 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.754743 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97a22f-0128-42fb-a735-c16c6f6c5b12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:36.784641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-0d97a22f-0128-42fb-a735-c16c6f6c5b12 None None] MFA Rules not processed for user `fc3f382490974ef1985df8878dd6a054`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:36.785848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0d97a22f-0128-42fb-a735-c16c6f6c5b12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:36.868681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1420/2842] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:36 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:36.884863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2064a1-520d-4f85-8c31-e26d0a6ad214 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:36.887141 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ac2064a1-520d-4f85-8c31-e26d0a6ad214 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:36.893228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:36.952663 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2064a1-520d-4f85-8c31-e26d0a6ad214 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:36.957100 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b71b31-dbe2-4109-8a41-f928030510e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:36.957712 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b71b31-dbe2-4109-8a41-f928030510e4 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:36.957989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b71b31-dbe2-4109-8a41-f928030510e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:36.958294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b71b31-dbe2-4109-8a41-f928030510e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:37.019090 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b71b31-dbe2-4109-8a41-f928030510e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:37.020527 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b71b31-dbe2-4109-8a41-f928030510e4 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:37.029479 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1423/2843] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:39.620868 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d97a22f-0128-42fb-a735-c16c6f6c5b12 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:39.633270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d97a22f-0128-42fb-a735-c16c6f6c5b12 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:39.637889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8397c44c-0644-42c5-a989-f2848cd825f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:39.638528 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8397c44c-0644-42c5-a989-f2848cd825f3 service cinder] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:39.638817 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8397c44c-0644-42c5-a989-f2848cd825f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:39.639074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8397c44c-0644-42c5-a989-f2848cd825f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:39.699003 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8397c44c-0644-42c5-a989-f2848cd825f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:39.700396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8397c44c-0644-42c5-a989-f2848cd825f3 service cinder] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:39.710391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1421/2844] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:39 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:40.282333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b71b31-dbe2-4109-8a41-f928030510e4 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:40.282333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-49b71b31-dbe2-4109-8a41-f928030510e4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:40.292571 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b71b31-dbe2-4109-8a41-f928030510e4 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:40.296485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ae5dab-1bed-4ea1-bf48-6f1bd6cac55a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:40.297092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae5dab-1bed-4ea1-bf48-6f1bd6cac55a service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:40.297690 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae5dab-1bed-4ea1-bf48-6f1bd6cac55a service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:40.298340 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ae5dab-1bed-4ea1-bf48-6f1bd6cac55a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:40.307495 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ae5dab-1bed-4ea1-bf48-6f1bd6cac55a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:40.308925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ae5dab-1bed-4ea1-bf48-6f1bd6cac55a service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:40.318865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1424/2845] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:40 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:41.479134 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8397c44c-0644-42c5-a989-f2848cd825f3 service cinder] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:41.480656 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8397c44c-0644-42c5-a989-f2848cd825f3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:41.547843 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8397c44c-0644-42c5-a989-f2848cd825f3 service cinder] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:41.552428 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:41.553057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:41.553331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:41.553709 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] PATH_INFO: `/v3/users/c46ec43e93f14ca7bf3cb87b006d832e` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:41.554848 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c46ec43e93f14ca7bf3cb87b006d832e)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:41.556791 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:41.559185 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:41.608517 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c46ec43e93f14ca7bf3cb87b006d832e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:41.609199 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c46ec43e93f14ca7bf3cb87b006d832e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:41.617067 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c46ec43e93f14ca7bf3cb87b006d832e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:41.617816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c46ec43e93f14ca7bf3cb87b006d832e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:41.626380 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c46ec43e93f14ca7bf3cb87b006d832e'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:41.650648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1422/2846] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:41 2022] DELETE /identity/v3/users/c46ec43e93f14ca7bf3cb87b006d832e => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:41.655256 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e38e2a-7e35-4bc8-ba47-07fa3c21a88e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:41.655760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e38e2a-7e35-4bc8-ba47-07fa3c21a88e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:41.655881 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e38e2a-7e35-4bc8-ba47-07fa3c21a88e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:41.684014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-53e38e2a-7e35-4bc8-ba47-07fa3c21a88e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:41.685488 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-53e38e2a-7e35-4bc8-ba47-07fa3c21a88e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:41.759009 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1425/2847] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:41.769333 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:41.770826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:41.830846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2346f866-a28d-4845-890a-b8445f697d7f admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:41.834943 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac8eb0-de24-4e12-b6c6-c1d196daae97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:41.835549 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac8eb0-de24-4e12-b6c6-c1d196daae97 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:41.835826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac8eb0-de24-4e12-b6c6-c1d196daae97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:41.836096 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac8eb0-de24-4e12-b6c6-c1d196daae97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:41.845156 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac8eb0-de24-4e12-b6c6-c1d196daae97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:41.846708 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac8eb0-de24-4e12-b6c6-c1d196daae97 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:41.856088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1423/2848] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:42.149164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e38e2a-7e35-4bc8-ba47-07fa3c21a88e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:42.150644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-53e38e2a-7e35-4bc8-ba47-07fa3c21a88e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:42.163189 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e38e2a-7e35-4bc8-ba47-07fa3c21a88e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:42.168162 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:42.168809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:42.169063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:42.169348 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] PATH_INFO: `/v3/projects/4c5c1cf00fe74cccb83f969e16eb0e5d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:42.170552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c5c1cf00fe74cccb83f969e16eb0e5d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:42.172416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:42.216989 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:42.225041 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c5c1cf00fe74cccb83f969e16eb0e5d', 'request_id': 'req-5a82ae42-b86e-4449-880f-52231e7aaf62'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.230742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c5c1cf00fe74cccb83f969e16eb0e5d', 'request_id': 'req-5a82ae42-b86e-4449-880f-52231e7aaf62'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.239307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c5c1cf00fe74cccb83f969e16eb0e5d', 'request_id': 'req-5a82ae42-b86e-4449-880f-52231e7aaf62'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.250644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] The token cache is being invalidate because project 4c5c1cf00fe74cccb83f969e16eb0e5d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:42.250939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.251834 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.253724 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1426/2849] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:42 2022] DELETE /identity/v3/projects/4c5c1cf00fe74cccb83f969e16eb0e5d => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:42.257942 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac8eb0-de24-4e12-b6c6-c1d196daae97 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:42.268354 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac8eb0-de24-4e12-b6c6-c1d196daae97 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:42.329484 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:42.330230 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:42.330417 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:42.330683 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] PATH_INFO: `/v3/users/0de5621b600740a6806ef40088b47305` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:42.331801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0de5621b600740a6806ef40088b47305)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:42.333714 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:42.364390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:42.386876 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0de5621b600740a6806ef40088b47305'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.387521 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0de5621b600740a6806ef40088b47305'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.394394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0de5621b600740a6806ef40088b47305'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.395122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0de5621b600740a6806ef40088b47305'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.404219 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f8a9bbb9-c4f5-438f-b2d0-1b6e2ad0e2e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0de5621b600740a6806ef40088b47305'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.421738 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1424/2850] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:42 2022] DELETE /identity/v3/users/0de5621b600740a6806ef40088b47305 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:42.711251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:42.712828 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:42.727253 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a82ae42-b86e-4449-880f-52231e7aaf62 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:42.789767 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:42.790434 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:42.790709 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:42.790975 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] PATH_INFO: `/v3/projects/f84e3d0ce448461dab8fa86512be157b` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:42.792140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f84e3d0ce448461dab8fa86512be157b)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:42.794030 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:42.844645 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f84e3d0ce448461dab8fa86512be157b', 'request_id': 'req-50357f54-b479-4728-aecf-847423f37699'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.850532 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f84e3d0ce448461dab8fa86512be157b', 'request_id': 'req-50357f54-b479-4728-aecf-847423f37699'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.859376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f84e3d0ce448461dab8fa86512be157b', 'request_id': 'req-50357f54-b479-4728-aecf-847423f37699'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.871872 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] The token cache is being invalidate because project f84e3d0ce448461dab8fa86512be157b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:42.872274 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.873191 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-50357f54-b479-4728-aecf-847423f37699 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:42.875042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1427/2851] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:42 2022] DELETE /identity/v3/projects/f84e3d0ce448461dab8fa86512be157b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:42.879908 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879ccd52-677a-45fc-b938-948afd4a6540 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:42.880394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879ccd52-677a-45fc-b938-948afd4a6540 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:42.880508 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879ccd52-677a-45fc-b938-948afd4a6540 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:42.901409 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-879ccd52-677a-45fc-b938-948afd4a6540 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:42.902631 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-879ccd52-677a-45fc-b938-948afd4a6540 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:42.966775 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1425/2852] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:42.976887 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f1f97-d3af-43bf-a597-3b2e67ea3e13 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:42.977289 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f1f97-d3af-43bf-a597-3b2e67ea3e13 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:42.977627 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f1f97-d3af-43bf-a597-3b2e67ea3e13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:43.000949 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-d64f1f97-d3af-43bf-a597-3b2e67ea3e13 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:43.002555 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d64f1f97-d3af-43bf-a597-3b2e67ea3e13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:43.010966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1428/2853] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:41:43.015322 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879ccd52-677a-45fc-b938-948afd4a6540 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:43.026227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879ccd52-677a-45fc-b938-948afd4a6540 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:43.031520 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f225a1e4-1af2-4c68-86d3-96ca8b48d95c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:43.032121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f225a1e4-1af2-4c68-86d3-96ca8b48d95c admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:43.032395 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f225a1e4-1af2-4c68-86d3-96ca8b48d95c admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:43.032642 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f225a1e4-1af2-4c68-86d3-96ca8b48d95c admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:43.033326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f225a1e4-1af2-4c68-86d3-96ca8b48d95c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:41:43.033620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f225a1e4-1af2-4c68-86d3-96ca8b48d95c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:43.035253 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f225a1e4-1af2-4c68-86d3-96ca8b48d95c admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:43.047397 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1426/2854] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:41:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:41:43.052091 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64f1f97-d3af-43bf-a597-3b2e67ea3e13 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:43.061960 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64f1f97-d3af-43bf-a597-3b2e67ea3e13 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:43.066757 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263a0a0b-522f-4d5e-ab69-228cfd3aa00d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:43.067359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263a0a0b-522f-4d5e-ab69-228cfd3aa00d admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:43.067796 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263a0a0b-522f-4d5e-ab69-228cfd3aa00d admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:43.067954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263a0a0b-522f-4d5e-ab69-228cfd3aa00d admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:43.068638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263a0a0b-522f-4d5e-ab69-228cfd3aa00d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:43.070568 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263a0a0b-522f-4d5e-ab69-228cfd3aa00d admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:43.103086 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1429/2855] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:43 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:41:43.107531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f225a1e4-1af2-4c68-86d3-96ca8b48d95c admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:43.120056 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f225a1e4-1af2-4c68-86d3-96ca8b48d95c admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:43.184426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dc88b8-0b5a-4843-becd-f8cfe46c4eff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:43.185049 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dc88b8-0b5a-4843-becd-f8cfe46c4eff admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:43.185334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dc88b8-0b5a-4843-becd-f8cfe46c4eff admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:43.185633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dc88b8-0b5a-4843-becd-f8cfe46c4eff admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:43.186339 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dc88b8-0b5a-4843-becd-f8cfe46c4eff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:43.188340 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dc88b8-0b5a-4843-becd-f8cfe46c4eff admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:43.215539 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1427/2856] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:43 2022] POST /identity/v3/users => generated 292 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:43.217062 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263a0a0b-522f-4d5e-ab69-228cfd3aa00d admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:43.227736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263a0a0b-522f-4d5e-ab69-228cfd3aa00d admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:43.232513 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2aa3b4-3e62-4c1f-a793-710e2632f22c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:43.233109 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2aa3b4-3e62-4c1f-a793-710e2632f22c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:43.233368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2aa3b4-3e62-4c1f-a793-710e2632f22c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:43.233636 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2aa3b4-3e62-4c1f-a793-710e2632f22c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:43.234303 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2aa3b4-3e62-4c1f-a793-710e2632f22c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:43.236002 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2aa3b4-3e62-4c1f-a793-710e2632f22c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:43.246795 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1430/2857] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:43.252963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dc88b8-0b5a-4843-becd-f8cfe46c4eff admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:43.262860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dc88b8-0b5a-4843-becd-f8cfe46c4eff admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:43.267357 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c452f9e-a20a-4552-bfe5-590b11a32b0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:43.267980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c452f9e-a20a-4552-bfe5-590b11a32b0d admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:43.268229 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c452f9e-a20a-4552-bfe5-590b11a32b0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:43.268498 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c452f9e-a20a-4552-bfe5-590b11a32b0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:43.269133 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c452f9e-a20a-4552-bfe5-590b11a32b0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:43.270773 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c452f9e-a20a-4552-bfe5-590b11a32b0d admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:43.281502 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1428/2858] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:43 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:43.286304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2aa3b4-3e62-4c1f-a793-710e2632f22c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:43.296414 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2aa3b4-3e62-4c1f-a793-710e2632f22c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:43.301218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eaabdb-55bd-4ff0-ad9c-098fbcdc8420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:43.301855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eaabdb-55bd-4ff0-ad9c-098fbcdc8420 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:43.302134 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eaabdb-55bd-4ff0-ad9c-098fbcdc8420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:43.302409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eaabdb-55bd-4ff0-ad9c-098fbcdc8420 admin admin] PATH_INFO: `/v3/projects/1a97b4ad6e4a4afdba6cb362f79aa10d/users/f774df8671274e8486ac209f4411efc9/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:43.317183 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09eaabdb-55bd-4ff0-ad9c-098fbcdc8420 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a97b4ad6e4a4afdba6cb362f79aa10d, user_id=f774df8671274e8486ac209f4411efc9, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:43.320765 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09eaabdb-55bd-4ff0-ad9c-098fbcdc8420 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:43.323140 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:43.338409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1431/2859] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:43 2022] PUT /identity/v3/projects/1a97b4ad6e4a4afdba6cb362f79aa10d/users/f774df8671274e8486ac209f4411efc9/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:43.349506 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc947943-49dd-4ab5-b900-163683d55c82 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:43.349792 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc947943-49dd-4ab5-b900-163683d55c82 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:43.350064 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc947943-49dd-4ab5-b900-163683d55c82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:43.371875 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:43.383376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-cc947943-49dd-4ab5-b900-163683d55c82 None None] MFA Rules not processed for user `f774df8671274e8486ac209f4411efc9`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:43.384999 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-cc947943-49dd-4ab5-b900-163683d55c82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:43.454973 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1429/2860] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:43 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:43.466749 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eaabdb-55bd-4ff0-ad9c-098fbcdc8420 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:43.468514 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-09eaabdb-55bd-4ff0-ad9c-098fbcdc8420 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:43.526282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eaabdb-55bd-4ff0-ad9c-098fbcdc8420 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:43.530183 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50de6a3-b805-466c-8000-de2936923b16 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:43.530786 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50de6a3-b805-466c-8000-de2936923b16 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:43.531063 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50de6a3-b805-466c-8000-de2936923b16 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:43.531333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50de6a3-b805-466c-8000-de2936923b16 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:43.543368 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50de6a3-b805-466c-8000-de2936923b16 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:43.545531 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50de6a3-b805-466c-8000-de2936923b16 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:43.555836 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1432/2861] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:44.263758 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc947943-49dd-4ab5-b900-163683d55c82 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:44.265317 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-cc947943-49dd-4ab5-b900-163683d55c82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:44.331802 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc947943-49dd-4ab5-b900-163683d55c82 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:44.340094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:44.340957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:44.341315 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:44.341632 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] PATH_INFO: `/v3/users/1257686c014a418098d97831461030be` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:44.343559 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1257686c014a418098d97831461030be)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:44.346352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:44.404248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1257686c014a418098d97831461030be'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:44.409250 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1257686c014a418098d97831461030be'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:44.417069 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1257686c014a418098d97831461030be'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:44.417801 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1257686c014a418098d97831461030be'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:44.431142 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1257686c014a418098d97831461030be'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:44.453592 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1430/2862] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:44 2022] DELETE /identity/v3/users/1257686c014a418098d97831461030be => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:44.458599 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9987a033-29a0-4896-a619-930cc52947f0 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:44.459114 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9987a033-29a0-4896-a619-930cc52947f0 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:44.459238 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9987a033-29a0-4896-a619-930cc52947f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:44.479292 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-9987a033-29a0-4896-a619-930cc52947f0 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:44.480504 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9987a033-29a0-4896-a619-930cc52947f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:44.553218 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1433/2863] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:44.563436 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:44.564961 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:44.586557 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:44.630639 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e918f357-388a-4613-9258-d8fbefd89722 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:44.635129 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23242583-7adf-4080-a3aa-e4dd873b1bbe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:44.635745 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23242583-7adf-4080-a3aa-e4dd873b1bbe service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:44.636027 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23242583-7adf-4080-a3aa-e4dd873b1bbe service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:44.636294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23242583-7adf-4080-a3aa-e4dd873b1bbe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:44.647061 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23242583-7adf-4080-a3aa-e4dd873b1bbe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:44.648562 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23242583-7adf-4080-a3aa-e4dd873b1bbe service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:44.658759 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1431/2864] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:45.222381 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9987a033-29a0-4896-a619-930cc52947f0 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:45.223877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9987a033-29a0-4896-a619-930cc52947f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:45.237667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9987a033-29a0-4896-a619-930cc52947f0 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:45.243084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:45.244199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:45.244485 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:45.245098 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] PATH_INFO: `/v3/projects/44c95c5ba3cf49e78c1c8a3f98a5752e` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:45.246480 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=44c95c5ba3cf49e78c1c8a3f98a5752e)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:45.248282 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:45.298378 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44c95c5ba3cf49e78c1c8a3f98a5752e', 'request_id': 'req-6011c292-26cf-46d5-b9f9-2ac36543a0f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.304298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44c95c5ba3cf49e78c1c8a3f98a5752e', 'request_id': 'req-6011c292-26cf-46d5-b9f9-2ac36543a0f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.313424 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44c95c5ba3cf49e78c1c8a3f98a5752e', 'request_id': 'req-6011c292-26cf-46d5-b9f9-2ac36543a0f8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.324863 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] The token cache is being invalidate because project 44c95c5ba3cf49e78c1c8a3f98a5752e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:45.325150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.325778 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.327294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1434/2865] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:45 2022] DELETE /identity/v3/projects/44c95c5ba3cf49e78c1c8a3f98a5752e => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:45.331765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23242583-7adf-4080-a3aa-e4dd873b1bbe service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:45.341957 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23242583-7adf-4080-a3aa-e4dd873b1bbe service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:45.400194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:45.400825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:45.401101 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:45.401365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] PATH_INFO: `/v3/users/8c7a720614434c72837d82cba2ef6cbc` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:45.402551 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c7a720614434c72837d82cba2ef6cbc)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:45.404480 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:45.453682 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c7a720614434c72837d82cba2ef6cbc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.454365 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c7a720614434c72837d82cba2ef6cbc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.461941 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c7a720614434c72837d82cba2ef6cbc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.462716 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c7a720614434c72837d82cba2ef6cbc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.472936 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c7a720614434c72837d82cba2ef6cbc'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.491093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1432/2866] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:45 2022] DELETE /identity/v3/users/8c7a720614434c72837d82cba2ef6cbc => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:45.775323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:45.776799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:45.792474 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6011c292-26cf-46d5-b9f9-2ac36543a0f8 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:45.811262 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:45.854702 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:45.855259 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:45.855536 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:45.855806 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] PATH_INFO: `/v3/projects/265aa1982e534368a6f631fce4915560` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:45.856964 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] RBAC: Authorizing `identity:delete_project(project_id=265aa1982e534368a6f631fce4915560)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:45.858905 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:45.907104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '265aa1982e534368a6f631fce4915560', 'request_id': 'req-5beed6d0-4284-485f-8046-8e950ec77b07'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.913222 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '265aa1982e534368a6f631fce4915560', 'request_id': 'req-5beed6d0-4284-485f-8046-8e950ec77b07'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.923879 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '265aa1982e534368a6f631fce4915560', 'request_id': 'req-5beed6d0-4284-485f-8046-8e950ec77b07'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.935436 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] The token cache is being invalidate because project 265aa1982e534368a6f631fce4915560 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:45.935785 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.936456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:45.937999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1435/2867] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:45 2022] DELETE /identity/v3/projects/265aa1982e534368a6f631fce4915560 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:45.942258 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:45.943796 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:45.957378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2d2575-5c47-4c3a-892b-ce6ce3f411b7 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:45.997091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:46.016964 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:46.017604 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:46.017883 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:46.018183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] PATH_INFO: `/v3/users/fc3f382490974ef1985df8878dd6a054` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:46.019414 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc3f382490974ef1985df8878dd6a054)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:46.021359 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:46.072509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc3f382490974ef1985df8878dd6a054'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:46.073243 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc3f382490974ef1985df8878dd6a054'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:46.080826 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc3f382490974ef1985df8878dd6a054'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:46.081485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc3f382490974ef1985df8878dd6a054'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:46.091582 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc3f382490974ef1985df8878dd6a054'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:46.109953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1433/2868] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:45 2022] DELETE /identity/v3/users/fc3f382490974ef1985df8878dd6a054 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:46.396938 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:46.398408 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:46.411921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5beed6d0-4284-485f-8046-8e950ec77b07 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:46.469940 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:46.470669 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:46.470939 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:46.471208 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] PATH_INFO: `/v3/projects/8b8310501ef74b3fa72e1d86e5ae97c6` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:46.472491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b8310501ef74b3fa72e1d86e5ae97c6)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:46.474507 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:46.522808 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b8310501ef74b3fa72e1d86e5ae97c6', 'request_id': 'req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:46.524684 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:46.529756 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b8310501ef74b3fa72e1d86e5ae97c6', 'request_id': 'req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:46.539737 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b8310501ef74b3fa72e1d86e5ae97c6', 'request_id': 'req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:46.552018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] The token cache is being invalidate because project 8b8310501ef74b3fa72e1d86e5ae97c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:46.552306 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:46.553033 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-a1c70504-b0c7-44b5-a99a-c2f489fcc1b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:46.554728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1436/2869] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:46 2022] DELETE /identity/v3/projects/8b8310501ef74b3fa72e1d86e5ae97c6 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:52.120294 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:52.122606 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:52.138723 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1569be-877b-4cdf-9a9b-e82c824c2011 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:52.197981 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:52.198641 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:52.198924 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:52.199196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] PATH_INFO: `/v3/users/f774df8671274e8486ac209f4411efc9` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:52.200321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f774df8671274e8486ac209f4411efc9)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:52.202335 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:52.256515 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f774df8671274e8486ac209f4411efc9'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:52.257171 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f774df8671274e8486ac209f4411efc9'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:52.263960 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f774df8671274e8486ac209f4411efc9'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:52.264589 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f774df8671274e8486ac209f4411efc9'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:52.268316 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:52.273638 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f774df8671274e8486ac209f4411efc9'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:52.292248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1434/2870] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:52 2022] DELETE /identity/v3/users/f774df8671274e8486ac209f4411efc9 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:52.296934 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41578c3d-ecd9-4388-9270-b46d348eca2b None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:52.297417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41578c3d-ecd9-4388-9270-b46d348eca2b None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:52.297545 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41578c3d-ecd9-4388-9270-b46d348eca2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:52.317956 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-41578c3d-ecd9-4388-9270-b46d348eca2b None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:52.319213 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-41578c3d-ecd9-4388-9270-b46d348eca2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:52.383723 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1437/2871] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:52.393825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:52.396121 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:52.453665 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c98e40-2962-4827-be28-086366ff82a8 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:52.457376 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a0c7c8-f56c-4c2b-950f-ac88c76dba5f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:52.458054 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a0c7c8-f56c-4c2b-950f-ac88c76dba5f service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:52.458352 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a0c7c8-f56c-4c2b-950f-ac88c76dba5f service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:52.458614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a0c7c8-f56c-4c2b-950f-ac88c76dba5f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:52.468698 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a0c7c8-f56c-4c2b-950f-ac88c76dba5f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:52.470135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a0c7c8-f56c-4c2b-950f-ac88c76dba5f service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:52.479378 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1435/2872] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:52.764042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41578c3d-ecd9-4388-9270-b46d348eca2b None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:52.765657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-41578c3d-ecd9-4388-9270-b46d348eca2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:52.780612 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41578c3d-ecd9-4388-9270-b46d348eca2b None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:52.786818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:52.787470 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:52.787742 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:52.788010 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] PATH_INFO: `/v3/projects/1a97b4ad6e4a4afdba6cb362f79aa10d` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:52.789307 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a97b4ad6e4a4afdba6cb362f79aa10d)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:52.791176 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:52.843999 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a97b4ad6e4a4afdba6cb362f79aa10d', 'request_id': 'req-523c8ba6-532a-4d48-a57f-aa70b29a0721'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:52.856093 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a97b4ad6e4a4afdba6cb362f79aa10d', 'request_id': 'req-523c8ba6-532a-4d48-a57f-aa70b29a0721'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:52.868301 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a97b4ad6e4a4afdba6cb362f79aa10d', 'request_id': 'req-523c8ba6-532a-4d48-a57f-aa70b29a0721'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:52.883760 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] The token cache is being invalidate because project 1a97b4ad6e4a4afdba6cb362f79aa10d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:52.884042 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:52.884736 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-523c8ba6-532a-4d48-a57f-aa70b29a0721 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:52.886335 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1438/2873] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:52 2022] DELETE /identity/v3/projects/1a97b4ad6e4a4afdba6cb362f79aa10d => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:52.891403 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e86818-f752-4b5a-a7ae-85c173276c40 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:52.891625 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e86818-f752-4b5a-a7ae-85c173276c40 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:52.891892 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e86818-f752-4b5a-a7ae-85c173276c40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:52.912505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-21e86818-f752-4b5a-a7ae-85c173276c40 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:52.913711 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-21e86818-f752-4b5a-a7ae-85c173276c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:52.977263 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1436/2874] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:52.985404 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1864c7e-ae47-402e-9616-3978fde9f7e9 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:52.985683 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1864c7e-ae47-402e-9616-3978fde9f7e9 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:52.985947 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1864c7e-ae47-402e-9616-3978fde9f7e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:53.007190 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-f1864c7e-ae47-402e-9616-3978fde9f7e9 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:53.008402 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-f1864c7e-ae47-402e-9616-3978fde9f7e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:53.016251 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1439/2875] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:53.020782 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e86818-f752-4b5a-a7ae-85c173276c40 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:53.030842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e86818-f752-4b5a-a7ae-85c173276c40 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:53.036392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18a07bc-ffeb-493f-81a0-1413e1021252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:53.036989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18a07bc-ffeb-493f-81a0-1413e1021252 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:53.037261 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18a07bc-ffeb-493f-81a0-1413e1021252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:53.037514 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18a07bc-ffeb-493f-81a0-1413e1021252 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:53.038194 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18a07bc-ffeb-493f-81a0-1413e1021252 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:41:53.038495 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18a07bc-ffeb-493f-81a0-1413e1021252 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:53.040094 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18a07bc-ffeb-493f-81a0-1413e1021252 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:53.052704 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1437/2876] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:41:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:41:53.056649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1864c7e-ae47-402e-9616-3978fde9f7e9 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:53.065991 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1864c7e-ae47-402e-9616-3978fde9f7e9 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:53.072061 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd3de04-9c34-4d65-aab2-fde765b25173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:53.072657 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd3de04-9c34-4d65-aab2-fde765b25173 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:53.072929 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd3de04-9c34-4d65-aab2-fde765b25173 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:53.073185 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd3de04-9c34-4d65-aab2-fde765b25173 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:53.073822 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd3de04-9c34-4d65-aab2-fde765b25173 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:53.075787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd3de04-9c34-4d65-aab2-fde765b25173 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:53.110747 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1440/2877] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:53 2022] POST /identity/v3/projects => generated 375 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:53.115035 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18a07bc-ffeb-493f-81a0-1413e1021252 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:53.125663 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18a07bc-ffeb-493f-81a0-1413e1021252 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:53.185135 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665b9312-b7f7-44c6-8bea-bdbfa7d1bdeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:53.185747 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665b9312-b7f7-44c6-8bea-bdbfa7d1bdeb admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:53.186024 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665b9312-b7f7-44c6-8bea-bdbfa7d1bdeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:53.186332 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665b9312-b7f7-44c6-8bea-bdbfa7d1bdeb admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:53.186954 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665b9312-b7f7-44c6-8bea-bdbfa7d1bdeb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:53.188890 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665b9312-b7f7-44c6-8bea-bdbfa7d1bdeb admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:53.206477 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:53.213278 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1438/2878] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:53 2022] POST /identity/v3/users => generated 288 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 13 15:41:53.217449 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd3de04-9c34-4d65-aab2-fde765b25173 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:53.228606 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd3de04-9c34-4d65-aab2-fde765b25173 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:53.233792 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2741321-52d8-41c9-8bf1-d3708513bdd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:53.234561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2741321-52d8-41c9-8bf1-d3708513bdd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:53.234837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2741321-52d8-41c9-8bf1-d3708513bdd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:53.235169 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2741321-52d8-41c9-8bf1-d3708513bdd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:53.235951 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2741321-52d8-41c9-8bf1-d3708513bdd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:53.237935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2741321-52d8-41c9-8bf1-d3708513bdd4 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:53.249353 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1441/2879] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:53 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:53.253634 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665b9312-b7f7-44c6-8bea-bdbfa7d1bdeb admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:53.264225 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665b9312-b7f7-44c6-8bea-bdbfa7d1bdeb admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:53.269762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6eb399e-9206-45eb-beab-1b5d6cfc1862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:53.270413 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eb399e-9206-45eb-beab-1b5d6cfc1862 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:53.270620 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eb399e-9206-45eb-beab-1b5d6cfc1862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:53.270948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eb399e-9206-45eb-beab-1b5d6cfc1862 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:53.271537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6eb399e-9206-45eb-beab-1b5d6cfc1862 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:53.273302 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6eb399e-9206-45eb-beab-1b5d6cfc1862 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:53.284574 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1439/2880] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:53.288846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2741321-52d8-41c9-8bf1-d3708513bdd4 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:53.299499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2741321-52d8-41c9-8bf1-d3708513bdd4 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:53.305710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8eca3f-0c0b-4c18-a9fe-60a9d7ef44b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:53.306400 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8eca3f-0c0b-4c18-a9fe-60a9d7ef44b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:53.306676 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8eca3f-0c0b-4c18-a9fe-60a9d7ef44b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:53.306941 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8eca3f-0c0b-4c18-a9fe-60a9d7ef44b0 admin admin] PATH_INFO: `/v3/projects/fa7f4f5d1cbc4bd68a3fbb629e1b41b4/users/2e89819f4b664708ac07acdb6c9071ec/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:53.322634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8eca3f-0c0b-4c18-a9fe-60a9d7ef44b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa7f4f5d1cbc4bd68a3fbb629e1b41b4, user_id=2e89819f4b664708ac07acdb6c9071ec, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:53.325300 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8eca3f-0c0b-4c18-a9fe-60a9d7ef44b0 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:53.337773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:53.342897 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1442/2881] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:53 2022] PUT /identity/v3/projects/fa7f4f5d1cbc4bd68a3fbb629e1b41b4/users/2e89819f4b664708ac07acdb6c9071ec/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:53.354627 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f943d8d7-d787-47a4-a295-d46fedf90422 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:53.354910 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f943d8d7-d787-47a4-a295-d46fedf90422 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:53.355203 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f943d8d7-d787-47a4-a295-d46fedf90422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:53.384869 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-f943d8d7-d787-47a4-a295-d46fedf90422 None None] MFA Rules not processed for user `2e89819f4b664708ac07acdb6c9071ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:53.386088 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f943d8d7-d787-47a4-a295-d46fedf90422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:53.456717 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1440/2882] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:53 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:53.471840 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8eca3f-0c0b-4c18-a9fe-60a9d7ef44b0 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:53.473661 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ab8eca3f-0c0b-4c18-a9fe-60a9d7ef44b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:53.539208 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8eca3f-0c0b-4c18-a9fe-60a9d7ef44b0 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:53.543029 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e2a775-f28c-49d5-a0a7-d39b6c2c555f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:53.543644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e2a775-f28c-49d5-a0a7-d39b6c2c555f service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:53.543921 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e2a775-f28c-49d5-a0a7-d39b6c2c555f service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:53.544219 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e2a775-f28c-49d5-a0a7-d39b6c2c555f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:53.552711 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e2a775-f28c-49d5-a0a7-d39b6c2c555f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:53.554159 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e2a775-f28c-49d5-a0a7-d39b6c2c555f service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:53.563066 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1443/2883] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:53 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:56.688093 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f943d8d7-d787-47a4-a295-d46fedf90422 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:56.758427 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f943d8d7-d787-47a4-a295-d46fedf90422 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:56.763309 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:56.763940 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:56.764220 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:56.764494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] PATH_INFO: `/v3/users/2e89819f4b664708ac07acdb6c9071ec` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:56.765652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e89819f4b664708ac07acdb6c9071ec)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:56.767608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:56.825568 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e89819f4b664708ac07acdb6c9071ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:56.826288 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e89819f4b664708ac07acdb6c9071ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:56.836394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e89819f4b664708ac07acdb6c9071ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:56.837115 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e89819f4b664708ac07acdb6c9071ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:56.848614 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e89819f4b664708ac07acdb6c9071ec'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:56.859648 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:56.867274 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1441/2884] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:56 2022] DELETE /identity/v3/users/2e89819f4b664708ac07acdb6c9071ec => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:56.872510 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d852caa-66dd-4afd-af3f-1fc156687e28 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:56.873018 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d852caa-66dd-4afd-af3f-1fc156687e28 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:56.873228 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d852caa-66dd-4afd-af3f-1fc156687e28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:56.892678 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-4d852caa-66dd-4afd-af3f-1fc156687e28 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:56.893875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4d852caa-66dd-4afd-af3f-1fc156687e28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:56.960237 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1444/2885] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:56.971401 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:56.972953 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:57.047489 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6822c70-90e0-4099-95b8-dd3d0d0ff643 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:57.051612 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0244bf-799e-412d-9d3c-9baf23d04014 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:57.052223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0244bf-799e-412d-9d3c-9baf23d04014 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.052499 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0244bf-799e-412d-9d3c-9baf23d04014 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.052767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0244bf-799e-412d-9d3c-9baf23d04014 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.062833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0244bf-799e-412d-9d3c-9baf23d04014 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:57.064283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0244bf-799e-412d-9d3c-9baf23d04014 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:57.073777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1442/2886] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:57.351069 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d852caa-66dd-4afd-af3f-1fc156687e28 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:57.352646 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-4d852caa-66dd-4afd-af3f-1fc156687e28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:57.369691 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d852caa-66dd-4afd-af3f-1fc156687e28 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:57.375181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:57.375829 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.376104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.376359 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] PATH_INFO: `/v3/projects/fa7f4f5d1cbc4bd68a3fbb629e1b41b4` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.377569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa7f4f5d1cbc4bd68a3fbb629e1b41b4)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:57.379694 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:57.430310 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa7f4f5d1cbc4bd68a3fbb629e1b41b4', 'request_id': 'req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:57.435911 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa7f4f5d1cbc4bd68a3fbb629e1b41b4', 'request_id': 'req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:57.444714 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa7f4f5d1cbc4bd68a3fbb629e1b41b4', 'request_id': 'req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:57.456188 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] The token cache is being invalidate because project fa7f4f5d1cbc4bd68a3fbb629e1b41b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:57.456472 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:57.457136 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-bb57569a-7ab7-49bd-9720-bf1b2e7c3d59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:57.458699 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1445/2887] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:57 2022] DELETE /identity/v3/projects/fa7f4f5d1cbc4bd68a3fbb629e1b41b4 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:57.465162 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5133a1-e3de-40ce-b1c0-d2c43dac1d3a None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.465326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5133a1-e3de-40ce-b1c0-d2c43dac1d3a None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.465608 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5133a1-e3de-40ce-b1c0-d2c43dac1d3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.486688 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-8a5133a1-e3de-40ce-b1c0-d2c43dac1d3a None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:57.487885 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8a5133a1-e3de-40ce-b1c0-d2c43dac1d3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:57.550246 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1443/2888] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:57.558464 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d694747-b333-4f14-a41c-af4e733a5c33 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.558752 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d694747-b333-4f14-a41c-af4e733a5c33 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.558992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d694747-b333-4f14-a41c-af4e733a5c33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.580422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-0d694747-b333-4f14-a41c-af4e733a5c33 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:57.581634 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0d694747-b333-4f14-a41c-af4e733a5c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:57.591419 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1446/2889] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:57.596762 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5133a1-e3de-40ce-b1c0-d2c43dac1d3a None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:57.612966 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5133a1-e3de-40ce-b1c0-d2c43dac1d3a None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:57.621390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc2f80b-679b-4fe1-ad66-2c9bdcde30c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:57.622011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc2f80b-679b-4fe1-ad66-2c9bdcde30c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.622315 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc2f80b-679b-4fe1-ad66-2c9bdcde30c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.622586 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc2f80b-679b-4fe1-ad66-2c9bdcde30c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.623238 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc2f80b-679b-4fe1-ad66-2c9bdcde30c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:41:57.623537 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc2f80b-679b-4fe1-ad66-2c9bdcde30c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:57.625132 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc2f80b-679b-4fe1-ad66-2c9bdcde30c7 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:57.638056 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1444/2890] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:41:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:41:57.642751 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d694747-b333-4f14-a41c-af4e733a5c33 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:57.652658 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d694747-b333-4f14-a41c-af4e733a5c33 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:57.658417 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3239f020-bb0e-432d-b574-e9d8ad850841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:57.659024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3239f020-bb0e-432d-b574-e9d8ad850841 admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.659298 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3239f020-bb0e-432d-b574-e9d8ad850841 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.659561 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3239f020-bb0e-432d-b574-e9d8ad850841 admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.660196 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3239f020-bb0e-432d-b574-e9d8ad850841 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:57.662132 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3239f020-bb0e-432d-b574-e9d8ad850841 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:57.685445 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:57.693456 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1447/2891] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:57 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:57.698478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc2f80b-679b-4fe1-ad66-2c9bdcde30c7 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:57.709579 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc2f80b-679b-4fe1-ad66-2c9bdcde30c7 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:57.771788 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c034aef-acf9-471c-aab9-c2949318d9b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:57.772338 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c034aef-acf9-471c-aab9-c2949318d9b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.772576 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c034aef-acf9-471c-aab9-c2949318d9b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.772866 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c034aef-acf9-471c-aab9-c2949318d9b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.773505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c034aef-acf9-471c-aab9-c2949318d9b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:57.775485 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c034aef-acf9-471c-aab9-c2949318d9b0 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:57.797582 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1445/2892] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:57 2022] POST /identity/v3/users => generated 287 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:57.801919 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3239f020-bb0e-432d-b574-e9d8ad850841 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:57.812686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3239f020-bb0e-432d-b574-e9d8ad850841 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:57.818016 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922da243-e1ad-4529-aa8e-092eb29a95f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:57.818643 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922da243-e1ad-4529-aa8e-092eb29a95f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.818913 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922da243-e1ad-4529-aa8e-092eb29a95f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.819199 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922da243-e1ad-4529-aa8e-092eb29a95f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.819885 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922da243-e1ad-4529-aa8e-092eb29a95f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:57.821469 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922da243-e1ad-4529-aa8e-092eb29a95f3 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:57.832187 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1448/2893] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 13 15:41:57.836652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c034aef-acf9-471c-aab9-c2949318d9b0 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:57.847392 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c034aef-acf9-471c-aab9-c2949318d9b0 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:57.852944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3a0305-11df-45ce-895c-2893f23fe216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:57.853545 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3a0305-11df-45ce-895c-2893f23fe216 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.853819 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3a0305-11df-45ce-895c-2893f23fe216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.854086 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3a0305-11df-45ce-895c-2893f23fe216 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.854765 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3a0305-11df-45ce-895c-2893f23fe216 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:57.856390 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3a0305-11df-45ce-895c-2893f23fe216 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:57.867509 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1446/2894] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:57.872092 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922da243-e1ad-4529-aa8e-092eb29a95f3 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:57.882704 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922da243-e1ad-4529-aa8e-092eb29a95f3 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:57.888909 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcd9ae4-c169-4ce3-a294-df91544e9c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:57.889559 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcd9ae4-c169-4ce3-a294-df91544e9c34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.889833 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcd9ae4-c169-4ce3-a294-df91544e9c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.890117 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcd9ae4-c169-4ce3-a294-df91544e9c34 admin admin] PATH_INFO: `/v3/projects/a8d664c8862f4e038672e343cab1d0bc/users/f107dc304fe7433ca98ae3761f8037a6/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.906589 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcd9ae4-c169-4ce3-a294-df91544e9c34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8d664c8862f4e038672e343cab1d0bc, user_id=f107dc304fe7433ca98ae3761f8037a6, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:57.909243 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcd9ae4-c169-4ce3-a294-df91544e9c34 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:57.924082 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1449/2895] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:57 2022] PUT /identity/v3/projects/a8d664c8862f4e038672e343cab1d0bc/users/f107dc304fe7433ca98ae3761f8037a6/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:57.935808 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ae1451-5183-44d5-9f73-4d86d66371e0 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:57.936091 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ae1451-5183-44d5-9f73-4d86d66371e0 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:57.936360 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ae1451-5183-44d5-9f73-4d86d66371e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:57.944531 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:57.965560 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-57ae1451-5183-44d5-9f73-4d86d66371e0 None None] MFA Rules not processed for user `f107dc304fe7433ca98ae3761f8037a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:57.966831 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-57ae1451-5183-44d5-9f73-4d86d66371e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:58.032355 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1447/2896] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:57 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:58.046980 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcd9ae4-c169-4ce3-a294-df91544e9c34 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:58.048846 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-ffcd9ae4-c169-4ce3-a294-df91544e9c34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:58.113130 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcd9ae4-c169-4ce3-a294-df91544e9c34 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:58.117224 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018521e0-059a-4fe3-b11f-ef19d20cc0c6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:58.117827 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018521e0-059a-4fe3-b11f-ef19d20cc0c6 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:58.118096 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018521e0-059a-4fe3-b11f-ef19d20cc0c6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:58.118386 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018521e0-059a-4fe3-b11f-ef19d20cc0c6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:58.126877 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018521e0-059a-4fe3-b11f-ef19d20cc0c6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:58.128296 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018521e0-059a-4fe3-b11f-ef19d20cc0c6 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:58.137172 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1450/2897] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:58 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:58.162724 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ae1451-5183-44d5-9f73-4d86d66371e0 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:58.228944 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ae1451-5183-44d5-9f73-4d86d66371e0 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:58.234597 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:58.235227 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:58.235501 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:58.235766 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] PATH_INFO: `/v3/users/f107dc304fe7433ca98ae3761f8037a6` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:58.237031 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] RBAC: Authorizing `identity:delete_user(user_id=f107dc304fe7433ca98ae3761f8037a6)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:58.239008 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:58.292548 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f107dc304fe7433ca98ae3761f8037a6'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:58.293127 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f107dc304fe7433ca98ae3761f8037a6'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:58.301284 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f107dc304fe7433ca98ae3761f8037a6'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:58.302011 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f107dc304fe7433ca98ae3761f8037a6'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:58.313781 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f107dc304fe7433ca98ae3761f8037a6'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:58.340193 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1448/2898] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:41:58 2022] DELETE /identity/v3/users/f107dc304fe7433ca98ae3761f8037a6 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:58.345060 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec9a96-3b6c-4829-8b0f-feaadde7f37e None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:58.345338 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec9a96-3b6c-4829-8b0f-feaadde7f37e None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:58.345620 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec9a96-3b6c-4829-8b0f-feaadde7f37e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:58.366886 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-a8ec9a96-3b6c-4829-8b0f-feaadde7f37e None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:58.368072 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a8ec9a96-3b6c-4829-8b0f-feaadde7f37e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:58.429769 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1451/2899] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:58.439878 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:58.441420 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:58.455877 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:58.503880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4310c6a5-449d-4a72-9fd4-be2c6166817e admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:58.508139 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6d9feb-7c2b-4c24-a31b-499858159179 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:58.508755 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6d9feb-7c2b-4c24-a31b-499858159179 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:58.509034 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6d9feb-7c2b-4c24-a31b-499858159179 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:58.509330 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6d9feb-7c2b-4c24-a31b-499858159179 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:58.518886 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6d9feb-7c2b-4c24-a31b-499858159179 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:58.520331 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6d9feb-7c2b-4c24-a31b-499858159179 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:58.530092 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1449/2900] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:41:58.813898 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec9a96-3b6c-4829-8b0f-feaadde7f37e None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:58.815447 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a8ec9a96-3b6c-4829-8b0f-feaadde7f37e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:58.827467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec9a96-3b6c-4829-8b0f-feaadde7f37e None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:58.832328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:58.832966 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:58.833242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:58.833508 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] PATH_INFO: `/v3/projects/a8d664c8862f4e038672e343cab1d0bc` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:58.834758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8d664c8862f4e038672e343cab1d0bc)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:58.836997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:58.886312 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8d664c8862f4e038672e343cab1d0bc', 'request_id': 'req-f2686679-37e0-41d4-b8d3-6d7f81895e3b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:58.891675 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8d664c8862f4e038672e343cab1d0bc', 'request_id': 'req-f2686679-37e0-41d4-b8d3-6d7f81895e3b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:58.894242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:58.902135 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8d664c8862f4e038672e343cab1d0bc', 'request_id': 'req-f2686679-37e0-41d4-b8d3-6d7f81895e3b'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:58.914293 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] The token cache is being invalidate because project a8d664c8862f4e038672e343cab1d0bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81455) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:41:58.914569 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:58.915276 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-f2686679-37e0-41d4-b8d3-6d7f81895e3b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:41:58.916865 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1452/2901] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:41:58 2022] DELETE /identity/v3/projects/a8d664c8862f4e038672e343cab1d0bc => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:58.922074 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff9419c-e145-4778-bce3-170ed34d7ed1 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:58.922596 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff9419c-e145-4778-bce3-170ed34d7ed1 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:58.922692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff9419c-e145-4778-bce3-170ed34d7ed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:58.942811 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-eff9419c-e145-4778-bce3-170ed34d7ed1 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:58.944748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-eff9419c-e145-4778-bce3-170ed34d7ed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:59.009276 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1450/2902] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:59.075328 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5a6f3-7811-4586-9f81-5a99e7aad275 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:59.075552 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5a6f3-7811-4586-9f81-5a99e7aad275 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:59.075841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5a6f3-7811-4586-9f81-5a99e7aad275 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:59.100164 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-7ab5a6f3-7811-4586-9f81-5a99e7aad275 None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:59.101413 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7ab5a6f3-7811-4586-9f81-5a99e7aad275 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:59.111728 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1453/2903] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:41:59.116833 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff9419c-e145-4778-bce3-170ed34d7ed1 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:59.128816 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff9419c-e145-4778-bce3-170ed34d7ed1 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:59.135186 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e4d44c-add5-48c1-9434-ce282b559d8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:59.135842 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4d44c-add5-48c1-9434-ce282b559d8b admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:59.136117 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4d44c-add5-48c1-9434-ce282b559d8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:59.136384 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e4d44c-add5-48c1-9434-ce282b559d8b admin admin] PATH_INFO: `/v3/domains` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:59.137026 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e4d44c-add5-48c1-9434-ce282b559d8b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81454) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 13 15:41:59.137326 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e4d44c-add5-48c1-9434-ce282b559d8b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:59.138967 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e4d44c-add5-48c1-9434-ce282b559d8b admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:59.151681 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1451/2904] 10.222.0.16 () {58 vars in 1153 bytes} [Tue Sep 13 15:41:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 13 15:41:59.155983 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab5a6f3-7811-4586-9f81-5a99e7aad275 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:59.165570 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab5a6f3-7811-4586-9f81-5a99e7aad275 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:59.171809 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c349fb59-e098-4d13-bcfe-953526ebe23c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:59.172410 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c349fb59-e098-4d13-bcfe-953526ebe23c admin admin] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:59.172682 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c349fb59-e098-4d13-bcfe-953526ebe23c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:59.172950 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c349fb59-e098-4d13-bcfe-953526ebe23c admin admin] PATH_INFO: `/v3/projects` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:59.173575 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c349fb59-e098-4d13-bcfe-953526ebe23c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:59.175524 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c349fb59-e098-4d13-bcfe-953526ebe23c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:59.207639 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1454/2905] 10.222.0.16 () {60 vars in 1153 bytes} [Tue Sep 13 15:41:59 2022] POST /identity/v3/projects => generated 381 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:59.212550 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e4d44c-add5-48c1-9434-ce282b559d8b admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:59.223124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e4d44c-add5-48c1-9434-ce282b559d8b admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:59.287909 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fddd76-d4f4-4ef1-8967-ec6441e12533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:59.288497 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fddd76-d4f4-4ef1-8967-ec6441e12533 admin admin] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:59.288770 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fddd76-d4f4-4ef1-8967-ec6441e12533 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:59.289037 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fddd76-d4f4-4ef1-8967-ec6441e12533 admin admin] PATH_INFO: `/v3/users` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:59.289673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fddd76-d4f4-4ef1-8967-ec6441e12533 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:59.291603 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fddd76-d4f4-4ef1-8967-ec6441e12533 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:59.315321 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1452/2906] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Sep 13 15:41:59 2022] POST /identity/v3/users => generated 291 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 13 15:41:59.319703 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c349fb59-e098-4d13-bcfe-953526ebe23c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:59.331104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c349fb59-e098-4d13-bcfe-953526ebe23c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:59.336894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33d452b-214c-4236-81e3-9d4acea23e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:59.337496 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33d452b-214c-4236-81e3-9d4acea23e3c admin admin] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:59.337784 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33d452b-214c-4236-81e3-9d4acea23e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:59.338040 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33d452b-214c-4236-81e3-9d4acea23e3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:59.338750 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33d452b-214c-4236-81e3-9d4acea23e3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:59.340837 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33d452b-214c-4236-81e3-9d4acea23e3c admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:59.350952 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1455/2907] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:59.355205 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fddd76-d4f4-4ef1-8967-ec6441e12533 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:59.365494 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fddd76-d4f4-4ef1-8967-ec6441e12533 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:59.370800 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ac4abf-c000-46b6-bb56-aea58ddc7594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:59.371396 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac4abf-c000-46b6-bb56-aea58ddc7594 admin admin] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:59.371674 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac4abf-c000-46b6-bb56-aea58ddc7594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:59.371948 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ac4abf-c000-46b6-bb56-aea58ddc7594 admin admin] PATH_INFO: `/v3/roles` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:59.372583 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ac4abf-c000-46b6-bb56-aea58ddc7594 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:59.374223 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ac4abf-c000-46b6-bb56-aea58ddc7594 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:59.384818 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1453/2908] 10.222.0.16 () {58 vars in 1122 bytes} [Tue Sep 13 15:41:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 13 15:41:59.389318 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33d452b-214c-4236-81e3-9d4acea23e3c admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:59.402205 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33d452b-214c-4236-81e3-9d4acea23e3c admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:59.409600 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79531920-432f-4edd-b53e-4b827c4ce6c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:59.410526 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79531920-432f-4edd-b53e-4b827c4ce6c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:59.411003 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79531920-432f-4edd-b53e-4b827c4ce6c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:59.411409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79531920-432f-4edd-b53e-4b827c4ce6c1 admin admin] PATH_INFO: `/v3/projects/cb52082b3d6d4b4aab5cc9b9c2e86cb3/users/2b9bd416dc3447509ab5451d2c0b479f/roles/7a7e816668a7493bad0752baca37934c` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:59.429818 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79531920-432f-4edd-b53e-4b827c4ce6c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb52082b3d6d4b4aab5cc9b9c2e86cb3, user_id=2b9bd416dc3447509ab5451d2c0b479f, role_id=7a7e816668a7493bad0752baca37934c)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:59.432625 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79531920-432f-4edd-b53e-4b827c4ce6c1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:59.446935 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1456/2909] 10.222.0.16 () {60 vars in 1483 bytes} [Tue Sep 13 15:41:59 2022] PUT /identity/v3/projects/cb52082b3d6d4b4aab5cc9b9c2e86cb3/users/2b9bd416dc3447509ab5451d2c0b479f/roles/7a7e816668a7493bad0752baca37934c => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:41:59.452488 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c55245-8ff7-4ee1-84f5-00cc78421ccd None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:59.452777 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c55245-8ff7-4ee1-84f5-00cc78421ccd None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:59.453036 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c55245-8ff7-4ee1-84f5-00cc78421ccd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:59.474557 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-63c55245-8ff7-4ee1-84f5-00cc78421ccd None None] MFA Rules not processed for user `d29c3b9b96c24a7daf5c64be528badeb`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:41:59.475782 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-63c55245-8ff7-4ee1-84f5-00cc78421ccd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:59.493591 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:41:59.539764 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1454/2910] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:41:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:41:59.550617 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79531920-432f-4edd-b53e-4b827c4ce6c1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:41:59.552844 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-79531920-432f-4edd-b53e-4b827c4ce6c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:41:59.619440 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79531920-432f-4edd-b53e-4b827c4ce6c1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:41:59.623540 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e22636c-6750-4898-803b-e8f266b0862a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:41:59.624151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22636c-6750-4898-803b-e8f266b0862a service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:41:59.624429 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22636c-6750-4898-803b-e8f266b0862a service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:41:59.624697 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e22636c-6750-4898-803b-e8f266b0862a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:41:59.636375 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e22636c-6750-4898-803b-e8f266b0862a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:41:59.637848 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e22636c-6750-4898-803b-e8f266b0862a service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:41:59.648638 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1457/2911] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:41:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:42:00.303754 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfa8558-baa5-4915-822f-968693f6c5d4 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:42:00.304264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfa8558-baa5-4915-822f-968693f6c5d4 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:42:00.304264 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfa8558-baa5-4915-822f-968693f6c5d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:42:00.333996 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-0dfa8558-baa5-4915-822f-968693f6c5d4 None None] MFA Rules not processed for user `2b9bd416dc3447509ab5451d2c0b479f`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:42:00.335248 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-0dfa8558-baa5-4915-822f-968693f6c5d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:42:00.407128 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1455/2912] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:42:00 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 13 15:42:00.418543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e22636c-6750-4898-803b-e8f266b0862a service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:42:00.420015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-9e22636c-6750-4898-803b-e8f266b0862a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:42:00.482849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e22636c-6750-4898-803b-e8f266b0862a service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:42:00.486741 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d39a74-8963-4e9b-b51e-7cbaafb03b21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '39c54901c5b5400f96e084fd4adf54e1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:42:00.487586 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d39a74-8963-4e9b-b51e-7cbaafb03b21 service cinder] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:42:00.487861 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d39a74-8963-4e9b-b51e-7cbaafb03b21 service cinder] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:42:00.488124 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d39a74-8963-4e9b-b51e-7cbaafb03b21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:42:00.497104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d39a74-8963-4e9b-b51e-7cbaafb03b21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:42:00.498566 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d39a74-8963-4e9b-b51e-7cbaafb03b21 service cinder] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:42:00.507292 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1458/2913] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:42:00 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:42:01.898402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a21f3f3-5fe7-40e1-acc8-70f212501d62 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:42:01.898963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a21f3f3-5fe7-40e1-acc8-70f212501d62 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:42:01.898963 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a21f3f3-5fe7-40e1-acc8-70f212501d62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:42:01.920169 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-3a21f3f3-5fe7-40e1-acc8-70f212501d62 None None] MFA Rules not processed for user `2b9bd416dc3447509ab5451d2c0b479f`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:42:01.921386 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3a21f3f3-5fe7-40e1-acc8-70f212501d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:42:01.929445 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1456/2914] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:42:01 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:42:02.174376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d39a74-8963-4e9b-b51e-7cbaafb03b21 service cinder] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:42:02.185415 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d39a74-8963-4e9b-b51e-7cbaafb03b21 service cinder] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:42:02.189324 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2776c7a3-575e-42b4-a6a3-bee4d877f148 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:42:02.189925 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776c7a3-575e-42b4-a6a3-bee4d877f148 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:42:02.190232 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776c7a3-575e-42b4-a6a3-bee4d877f148 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:42:02.190499 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776c7a3-575e-42b4-a6a3-bee4d877f148 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:42:02.192314 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2776c7a3-575e-42b4-a6a3-bee4d877f148 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:42:02.207667 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2776c7a3-575e-42b4-a6a3-bee4d877f148 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:42:02.209821 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2776c7a3-575e-42b4-a6a3-bee4d877f148 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:42:02.220271 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1459/2915] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:42:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:42:02.572452 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a21f3f3-5fe7-40e1-acc8-70f212501d62 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:42:02.574057 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-3a21f3f3-5fe7-40e1-acc8-70f212501d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:42:02.646426 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a21f3f3-5fe7-40e1-acc8-70f212501d62 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:42:02.651254 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b0b55c-c6e5-4680-84b7-1061bc8fb784 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:42:02.652077 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b0b55c-c6e5-4680-84b7-1061bc8fb784 service placement] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:42:02.652510 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b0b55c-c6e5-4680-84b7-1061bc8fb784 service placement] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:42:02.652785 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b0b55c-c6e5-4680-84b7-1061bc8fb784 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:42:02.654558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-95b0b55c-c6e5-4680-84b7-1061bc8fb784 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:42:02.665742 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b0b55c-c6e5-4680-84b7-1061bc8fb784 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:42:02.667391 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b0b55c-c6e5-4680-84b7-1061bc8fb784 service placement] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:42:02.677042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1457/2916] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:42:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:42:10.712166 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2776c7a3-575e-42b4-a6a3-bee4d877f148 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:42:10.714128 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-2776c7a3-575e-42b4-a6a3-bee4d877f148 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:42:10.736425 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:DELETE=1,keystone:SELECT=97,keystone:INSERT=2 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:42:10.797649 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2776c7a3-575e-42b4-a6a3-bee4d877f148 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:42:10.803216 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae78688-54bb-47ae-9fe0-c8dd51f4d148 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:42:10.803855 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae78688-54bb-47ae-9fe0-c8dd51f4d148 service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:42:10.804126 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae78688-54bb-47ae-9fe0-c8dd51f4d148 service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:42:10.804388 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae78688-54bb-47ae-9fe0-c8dd51f4d148 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:42:10.806242 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-bae78688-54bb-47ae-9fe0-c8dd51f4d148 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:42:10.818995 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae78688-54bb-47ae-9fe0-c8dd51f4d148 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:42:10.820577 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae78688-54bb-47ae-9fe0-c8dd51f4d148 service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:42:10.831710 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1460/2917] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:42:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:42:12.672947 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=84 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:42:20.827332 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=26 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:43:17.516899 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76f51ad-6606-4eb9-bd13-fbc318cdfe20 None None] REQUEST_METHOD: `POST` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:43:17.517461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76f51ad-6606-4eb9-bd13-fbc318cdfe20 None None] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:43:17.517461 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76f51ad-6606-4eb9-bd13-fbc318cdfe20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:43:17.538529 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.auth.core [None req-e76f51ad-6606-4eb9-bd13-fbc318cdfe20 None None] MFA Rules not processed for user `2b9bd416dc3447509ab5451d2c0b479f`. Rule list: `[]` (Enabled: `True`). {{(pid=81454) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:43:17.539767 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-e76f51ad-6606-4eb9-bd13-fbc318cdfe20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:43:17.547727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1458/2918] 10.222.0.16 () {58 vars in 958 bytes} [Tue Sep 13 15:43:17 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:43:19.533347 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae78688-54bb-47ae-9fe0-c8dd51f4d148 service nova] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:43:19.534725 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-bae78688-54bb-47ae-9fe0-c8dd51f4d148 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:43:19.550418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae78688-54bb-47ae-9fe0-c8dd51f4d148 service nova] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:43:19.555633 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49205e06-dc8a-4b04-92c4-254ba5420e3f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:43:19.556249 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49205e06-dc8a-4b04-92c4-254ba5420e3f service placement] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:43:19.556519 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49205e06-dc8a-4b04-92c4-254ba5420e3f service placement] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:43:19.556789 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49205e06-dc8a-4b04-92c4-254ba5420e3f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:43:19.558859 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-49205e06-dc8a-4b04-92c4-254ba5420e3f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:43:19.571263 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49205e06-dc8a-4b04-92c4-254ba5420e3f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:43:19.573317 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49205e06-dc8a-4b04-92c4-254ba5420e3f service placement] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:43:19.583294 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1461/2919] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:43:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:43:27.533081 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=3 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:43:29.579058 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=6 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:44:23.850447 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76f51ad-6606-4eb9-bd13-fbc318cdfe20 None None] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:44:23.863124 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76f51ad-6606-4eb9-bd13-fbc318cdfe20 None None] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:44:23.868664 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280c64d4-43e7-42a3-b8e8-a86a1ab52e36 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:44:23.869191 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280c64d4-43e7-42a3-b8e8-a86a1ab52e36 service placement] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:44:23.869464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280c64d4-43e7-42a3-b8e8-a86a1ab52e36 service placement] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:44:23.869729 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280c64d4-43e7-42a3-b8e8-a86a1ab52e36 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:44:23.871536 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-280c64d4-43e7-42a3-b8e8-a86a1ab52e36 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:44:23.885234 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280c64d4-43e7-42a3-b8e8-a86a1ab52e36 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:44:23.887418 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280c64d4-43e7-42a3-b8e8-a86a1ab52e36 service placement] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:44:23.900956 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1459/2920] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:44:23 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:44:23.931992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49205e06-dc8a-4b04-92c4-254ba5420e3f service placement] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:44:23.944845 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49205e06-dc8a-4b04-92c4-254ba5420e3f service placement] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:44:23.950894 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:44:23.951919 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:44:23.952416 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:44:23.952787 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] PATH_INFO: `/v3/users/2b9bd416dc3447509ab5451d2c0b479f` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:44:23.954376 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b9bd416dc3447509ab5451d2c0b479f)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:44:23.956422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:44:24.007423 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b9bd416dc3447509ab5451d2c0b479f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:44:24.016333 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b9bd416dc3447509ab5451d2c0b479f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:44:24.016954 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b9bd416dc3447509ab5451d2c0b479f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:44:24.017860 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b9bd416dc3447509ab5451d2c0b479f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:44:24.024930 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b9bd416dc3447509ab5451d2c0b479f'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:44:24.042461 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1462/2921] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:44:23 2022] DELETE /identity/v3/users/2b9bd416dc3447509ab5451d2c0b479f => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:44:24.230075 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280c64d4-43e7-42a3-b8e8-a86a1ab52e36 service placement] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:44:24.242860 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280c64d4-43e7-42a3-b8e8-a86a1ab52e36 service placement] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:44:24.302554 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:44:24.303196 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:44:24.303475 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:44:24.303748 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] PATH_INFO: `/v3/projects/cb52082b3d6d4b4aab5cc9b9c2e86cb3` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:44:24.304880 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb52082b3d6d4b4aab5cc9b9c2e86cb3)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:44:24.306844 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:44:24.358439 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb52082b3d6d4b4aab5cc9b9c2e86cb3', 'request_id': 'req-fb160173-69e0-4ddf-92f9-b641ae663573'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:44:24.365089 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb52082b3d6d4b4aab5cc9b9c2e86cb3', 'request_id': 'req-fb160173-69e0-4ddf-92f9-b641ae663573'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:44:24.376611 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb52082b3d6d4b4aab5cc9b9c2e86cb3', 'request_id': 'req-fb160173-69e0-4ddf-92f9-b641ae663573'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:44:24.388884 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] The token cache is being invalidate because project cb52082b3d6d4b4aab5cc9b9c2e86cb3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:44:24.389163 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:44:24.389863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:44:24.391564 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1460/2922] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:44:24 2022] DELETE /identity/v3/projects/cb52082b3d6d4b4aab5cc9b9c2e86cb3 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:44:34.039181 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=29,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:44:34.385728 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=57,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:45:10.839186 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:45:10.841074 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:45:10.904011 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b81c64-4f38-4420-a6c7-0053dc9be132 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:45:10.956866 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0e67f0-7855-4171-8930-9758ce5959c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:45:10.957477 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0e67f0-7855-4171-8930-9758ce5959c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:45:10.957748 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0e67f0-7855-4171-8930-9758ce5959c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:45:10.958014 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0e67f0-7855-4171-8930-9758ce5959c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:45:10.959997 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7c0e67f0-7855-4171-8930-9758ce5959c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:45:11.021409 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0e67f0-7855-4171-8930-9758ce5959c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:45:11.022874 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0e67f0-7855-4171-8930-9758ce5959c3 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:45:11.033279 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1463/2923] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:45:10 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:45:12.359072 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:45:12.360654 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:45:12.373408 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb160173-69e0-4ddf-92f9-b641ae663573 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:45:12.377014 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c98d2-d6d2-4d75-adc0-e96b073d1cf4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:45:12.377617 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c98d2-d6d2-4d75-adc0-e96b073d1cf4 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:45:12.377889 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c98d2-d6d2-4d75-adc0-e96b073d1cf4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:45:12.378167 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c98d2-d6d2-4d75-adc0-e96b073d1cf4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:45:12.379854 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-8a8c98d2-d6d2-4d75-adc0-e96b073d1cf4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:45:12.454929 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c98d2-d6d2-4d75-adc0-e96b073d1cf4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:45:12.456727 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c98d2-d6d2-4d75-adc0-e96b073d1cf4 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:45:12.469649 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1461/2924] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:45:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:45:14.408104 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0e67f0-7855-4171-8930-9758ce5959c3 service neutron] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:45:14.409835 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-7c0e67f0-7855-4171-8930-9758ce5959c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:45:14.428015 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0e67f0-7855-4171-8930-9758ce5959c3 service neutron] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:45:14.433884 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b806e654-f62b-4b5f-b398-1389a3fe083a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f027e913f53447a988f74ae4dff84b2c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:45:14.434538 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806e654-f62b-4b5f-b398-1389a3fe083a service nova] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:45:14.434815 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806e654-f62b-4b5f-b398-1389a3fe083a service nova] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:45:14.435084 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806e654-f62b-4b5f-b398-1389a3fe083a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:45:14.436801 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b806e654-f62b-4b5f-b398-1389a3fe083a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:45:14.449771 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b806e654-f62b-4b5f-b398-1389a3fe083a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:45:14.451431 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b806e654-f62b-4b5f-b398-1389a3fe083a service nova] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:45:14.462799 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1464/2925] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:45:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:45:16.243435 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c98d2-d6d2-4d75-adc0-e96b073d1cf4 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:45:16.253032 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c98d2-d6d2-4d75-adc0-e96b073d1cf4 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:45:16.257334 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb063945-3a7b-4f50-af18-61018241b998 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:45:16.257989 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb063945-3a7b-4f50-af18-61018241b998 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:45:16.258299 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb063945-3a7b-4f50-af18-61018241b998 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:45:16.258573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb063945-3a7b-4f50-af18-61018241b998 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:45:16.261126 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-fb063945-3a7b-4f50-af18-61018241b998 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:45:16.323994 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb063945-3a7b-4f50-af18-61018241b998 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:45:16.325481 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb063945-3a7b-4f50-af18-61018241b998 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:45:16.335277 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1462/2926] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:45:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:45:22.619992 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b806e654-f62b-4b5f-b398-1389a3fe083a service nova] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:45:22.621670 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-b806e654-f62b-4b5f-b398-1389a3fe083a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:45:22.637543 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b806e654-f62b-4b5f-b398-1389a3fe083a service nova] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:45:22.643089 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:45:22.643737 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:45:22.644012 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:45:22.644283 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] PATH_INFO: `/v3/users/cb98650b97664e0cbac0f574163b8c40` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:45:22.645450 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb98650b97664e0cbac0f574163b8c40)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:45:22.647370 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:45:22.697245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb98650b97664e0cbac0f574163b8c40'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:22.700745 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:45:22.707455 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb98650b97664e0cbac0f574163b8c40'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:22.708374 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb98650b97664e0cbac0f574163b8c40'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:22.709518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb98650b97664e0cbac0f574163b8c40'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:22.717422 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb98650b97664e0cbac0f574163b8c40'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:22.739403 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1465/2927] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:45:22 2022] DELETE /identity/v3/users/cb98650b97664e0cbac0f574163b8c40 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:45:22.924872 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb063945-3a7b-4f50-af18-61018241b998 service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:45:22.935847 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb063945-3a7b-4f50-af18-61018241b998 service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:45:22.994997 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:45:22.995633 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:45:22.995915 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:45:22.996183 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] PATH_INFO: `/v3/projects/7eb316ace93e48c68ade74e9c5fccbf8` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:45:22.997846 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7eb316ace93e48c68ade74e9c5fccbf8)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:45:22.999828 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:45:23.021998 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=100 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:45:23.051104 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7eb316ace93e48c68ade74e9c5fccbf8', 'request_id': 'req-47458670-d330-4091-a5f2-5ecb2b971739'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.057300 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7eb316ace93e48c68ade74e9c5fccbf8', 'request_id': 'req-47458670-d330-4091-a5f2-5ecb2b971739'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.066451 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7eb316ace93e48c68ade74e9c5fccbf8', 'request_id': 'req-47458670-d330-4091-a5f2-5ecb2b971739'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.078240 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] The token cache is being invalidate because project 7eb316ace93e48c68ade74e9c5fccbf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:45:23.078523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.079160 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.080657 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1463/2928] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:45:22 2022] DELETE /identity/v3/projects/7eb316ace93e48c68ade74e9c5fccbf8 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 13 15:45:23.084923 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:45:23.086664 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:45:23.101034 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59b7d8a-82b1-4fdb-9e07-13394fb70ee1 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:45:23.162849 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:45:23.163487 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:45:23.163755 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:45:23.164024 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] PATH_INFO: `/v3/users/49d56d51ce70408ab780ef7c6faba482` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:45:23.165334 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] RBAC: Authorizing `identity:delete_user(user_id=49d56d51ce70408ab780ef7c6faba482)` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:45:23.167297 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:45:23.221463 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49d56d51ce70408ab780ef7c6faba482'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.230875 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49d56d51ce70408ab780ef7c6faba482'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.231644 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49d56d51ce70408ab780ef7c6faba482'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.232679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49d56d51ce70408ab780ef7c6faba482'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.240149 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.notifications [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49d56d51ce70408ab780ef7c6faba482'} {{(pid=81455) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.257442 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1466/2929] 10.222.0.16 () {58 vars in 1224 bytes} [Tue Sep 13 15:45:23 2022] DELETE /identity/v3/users/49d56d51ce70408ab780ef7c6faba482 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:45:23.449154 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:45:23.451558 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:45:23.465980 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47458670-d330-4091-a5f2-5ecb2b971739 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:45:23.526505 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd29c3b9b96c24a7daf5c64be528badeb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3dcdfc529b6149688ceee6b949a735a1', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:45:23.527164 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:45:23.527441 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:45:23.527710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] PATH_INFO: `/v3/projects/c4845b9e9baf48ffb49701d088fb2400` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:45:23.529394 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4845b9e9baf48ffb49701d088fb2400)` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:45:23.531368 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:45:23.583179 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4845b9e9baf48ffb49701d088fb2400', 'request_id': 'req-f6873077-f2c4-499f-84c4-59faeac84191'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.588673 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4845b9e9baf48ffb49701d088fb2400', 'request_id': 'req-f6873077-f2c4-499f-84c4-59faeac84191'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.597669 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4845b9e9baf48ffb49701d088fb2400', 'request_id': 'req-f6873077-f2c4-499f-84c4-59faeac84191'} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.610283 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] The token cache is being invalidate because project c4845b9e9baf48ffb49701d088fb2400 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81454) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 13 15:45:23.610573 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.611299 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.notifications [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81454) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 13 15:45:23.612863 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1464/2930] 10.222.0.16 () {58 vars in 1233 bytes} [Tue Sep 13 15:45:23 2022] DELETE /identity/v3/projects/c4845b9e9baf48ffb49701d088fb2400 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 13 15:45:33.254356 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=65,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:45:33.607270 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:DELETE=14,keystone:SELECT=66,keystone:INSERT=2 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:47:56.677758 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:47:56.679475 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:47:56.743775 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d54c497-3ae4-43eb-923a-aca9cc385652 admin admin] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:47:56.804323 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798bb806-2253-478e-98ae-edde7c46a355 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:47:56.805150 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798bb806-2253-478e-98ae-edde7c46a355 service neutron] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:47:56.805491 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798bb806-2253-478e-98ae-edde7c46a355 service neutron] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:47:56.805841 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798bb806-2253-478e-98ae-edde7c46a355 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:47:56.808418 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-798bb806-2253-478e-98ae-edde7c46a355 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:47:56.886910 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798bb806-2253-478e-98ae-edde7c46a355 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:47:56.888686 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798bb806-2253-478e-98ae-edde7c46a355 service neutron] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:47:56.900151 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1467/2931] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:47:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:47:56.908297 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:47:56.910028 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:47:56.923692 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6873077-f2c4-499f-84c4-59faeac84191 admin admin] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:47:56.928710 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8e4d42-0137-410c-b865-f1e17ff4813e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:47:56.929405 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8e4d42-0137-410c-b865-f1e17ff4813e service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:47:56.929825 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8e4d42-0137-410c-b865-f1e17ff4813e service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:47:56.930212 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8e4d42-0137-410c-b865-f1e17ff4813e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:47:56.932464 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.fernet_utils [None req-9f8e4d42-0137-410c-b865-f1e17ff4813e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81454) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application [None req-9f8e4d42-0137-410c-b865-f1e17ff4813e service neutron] Could not find project: f99e1e9c3c13491e86734af2b526fc42.: keystone.exception.ProjectNotFound: Could not find project: f99e1e9c3c13491e86734af2b526fc42. Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Sep 13 15:47:56.973251 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:47:56.975042 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self._validate_token_resources() Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 13 15:47:56.976652 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.get_or_create( Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application with Lock( Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._enter() Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.creator() Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application created_value = creator( Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 13 15:47:56.978478 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 13 15:47:56.979995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: f99e1e9c3c13491e86734af2b526fc42. Sep 13 15:47:56.979995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: ERROR keystone.server.flask.application Sep 13 15:47:56.979995 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1465/2932] 10.222.0.16 () {60 vars in 1396 bytes} [Tue Sep 13 15:47:56 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 113 bytes in 68 msecs (HTTP/1.1 404) 5 headers in 184 bytes (1 switches on core 0) Sep 13 15:47:57.015962 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58725b2e-52b4-40a5-9281-5322bbc0f704 None None] REQUEST_METHOD: `POST` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:47:57.016245 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58725b2e-52b4-40a5-9281-5322bbc0f704 None None] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:47:57.016518 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58725b2e-52b4-40a5-9281-5322bbc0f704 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:47:57.060467 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.auth.core [None req-58725b2e-52b4-40a5-9281-5322bbc0f704 None None] MFA Rules not processed for user `f027e913f53447a988f74ae4dff84b2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81455) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 13 15:47:57.061695 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-58725b2e-52b4-40a5-9281-5322bbc0f704 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:47:57.076344 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1468/2933] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Sep 13 15:47:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 13 15:47:57.088496 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8e4d42-0137-410c-b865-f1e17ff4813e service neutron] Authenticating user token {{(pid=81454) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:47:57.098570 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8e4d42-0137-410c-b865-f1e17ff4813e service neutron] Validating token access rules against request {{(pid=81454) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:47:57.103523 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6bde85-efd0-46ab-b06c-98315c337712 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '171e916244804add8f18b44b77b03d72', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81454) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:47:57.104122 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6bde85-efd0-46ab-b06c-98315c337712 service neutron] REQUEST_METHOD: `GET` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:47:57.104402 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6bde85-efd0-46ab-b06c-98315c337712 service neutron] SCRIPT_NAME: `/identity` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:47:57.104671 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6bde85-efd0-46ab-b06c-98315c337712 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81454) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:47:57.116008 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6bde85-efd0-46ab-b06c-98315c337712 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:47:57.117468 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6bde85-efd0-46ab-b06c-98315c337712 service neutron] RBAC: Authorization granted {{(pid=81454) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:47:57.128347 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: [pid: 81454|app: 0|req: 1466/2934] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:47:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:48:07.057773 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=83 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:48:07.123746 dv-cinder-857434-2-iscsi devstack@keystone.service[81454]: DEBUG dbcounter [-] [81454] Writing DB stats keystone:SELECT=10 {{(pid=81454) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 13 15:49:16.659144 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58725b2e-52b4-40a5-9281-5322bbc0f704 None None] Authenticating user token {{(pid=81455) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 13 15:49:16.660732 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-58725b2e-52b4-40a5-9281-5322bbc0f704 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:49:16.829562 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58725b2e-52b4-40a5-9281-5322bbc0f704 None None] Validating token access rules against request {{(pid=81455) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 13 15:49:16.835679 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a413af70-4297-4506-a104-83d3e669b3d7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4bc6be7644b24ca9bab23128b41a7148', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c30b8f78ae8c4ee2b59357d0be54f953', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81455) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 13 15:49:16.836285 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a413af70-4297-4506-a104-83d3e669b3d7 service placement] REQUEST_METHOD: `GET` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 13 15:49:16.836557 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a413af70-4297-4506-a104-83d3e669b3d7 service placement] SCRIPT_NAME: `/identity` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 13 15:49:16.836819 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a413af70-4297-4506-a104-83d3e669b3d7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81455) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 13 15:49:16.838744 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.fernet_utils [None req-a413af70-4297-4506-a104-83d3e669b3d7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81455) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 13 15:49:16.850304 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a413af70-4297-4506-a104-83d3e669b3d7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 13 15:49:16.851857 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a413af70-4297-4506-a104-83d3e669b3d7 service placement] RBAC: Authorization granted {{(pid=81455) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 13 15:49:16.862636 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: [pid: 81455|app: 0|req: 1469/2935] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Sep 13 15:49:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 13 15:49:26.858083 dv-cinder-857434-2-iscsi devstack@keystone.service[81455]: DEBUG dbcounter [-] [81455] Writing DB stats keystone:SELECT=63 {{(pid=81455) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}